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1  Introduction 


The  U.S.  Department  of  the  Navy  has  a  number  of  challenges  in  facilitating  a  high  level 
of  military  readiness.  Currently,  operational  commanders  and  policymakers  examine  ratings  on 
the  Status  of  Training  and  Resources  System  (SORTS)  as  a  readiness  metric.  SORTS  generally 
consists  of  three  primary  military  readiness  factors — manning  levels,  equipment  and  supplies, 
and  completed  training  evolutions.  Clearly,  Sailor  retention  and  training  play  key  roles  in 
developing  and  maintaining  military  readiness. 

The  Chief  of  Naval  Operations  (CNO)  chartered  the  Executive  Review  of  Navy  Training 
(2001),  which  subsequently  led  to  the  formation  of  a  Task  Force  for  Excellence  through 
Commitment  to  Education  and  Learning  (EXCEL).  Task  Force  EXCEL’s  goal  was  to  identify 
new  ways  for  the  U.S.  Navy  to  train,  grow,  place,  and  utilize  personnel  who  maximize  the 
Navy’s  ability  to  accomplish  its  military  mission  while  developing  a  more  productive  yet 
satisfying  workplace. 

Task  Force  EXCEL  consists  of  five  components  or  “ vectors ”  that  are  essential  to  how 
personnel  meet  their  missions  and  manage  the  Navy  workforce.  These  five  vectors  include 
Professional  Development,  Personal  Development,  Professional  Military  Education  and 
Leadership,  Certifications  and  Qualifications,  and  Performance.  The  primary  tasking  of  the 
Performance  Vector  includes  an  examination  of  the  Navy  performance  appraisal  and 
management  system. 

One  challenge  for  the  Performance  Vector  was  the  need  for  a  performance  appraisal  and 
management  system  that  is  aligned  with  the  changing  workplace  performance  needs  of  the  U.S. 
Navy.  Since  1996,  the  Navy  has  operated  with  a  trait-based  performance  appraisal  system,  where 
supervisors  have  rated  personnel  on  traits  such  as  leadership,  teamwork,  equal  opportunity,  and 
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military  bearing/character  (BUPERSINST  1600.10,  1995).  One  recommendation  from  initial 
Task  Force  EXCEL  meetings  was  a  behaviorally  based  performance  appraisal  system. 
Additionally,  after  examining  military  and  civilian  best  practices  in  performance  appraisal  and 
management,  and  learning  of  the  CNO’s  desire  for  an  electronically  based  performance 
management/appraisal  system,  the  Performance  Vector  recommended  the  development  of  a 
behaviorally  based  performance  appraisal  system. 

As  the  Commander,  Navy  Personnel  Command  (CNPC)  develops  a  new  performance 
management  and  appraisal  system,  it  is  confronted  with  having  to  develop  performance  appraisal 
systems  that  are  fully  operational  and  integrated  with  the  performance  evaluation  and  promotion 
selection  cycle.  As  the  new  Human  Performance  Feedback  and  Development  (HPFD) 
performance  management  and  appraisal  system  is  implemented,  the  final  performance  appraisal 
forms  as  formatted  and  presented  in  the  PeopleSoft  8.8  (2004)  application  require  usability 
testing  with  supervisory  and  non- supervisory  Navy  personnel  to  identify  usability  concerns  and 
improve  the  functionality  of  the  electronically  based  performance  management  and  appraisal 
system.  Usability  testing  is  a  vital  step  in  the  development  of  any  new  Web-based  tool.  In  theory, 
the  automated  tool  should  reduce  the  burden  on  users;  in  practice,  however,  such  tools  can  be 
more  difficult  to  figure  out  than  their  paper  counterparts.  Usability  testing  can  assess  the  time  it 
takes  to  complete  a  form,  the  amount  of  self-editing  required,  and  the  navigational  problems 
users  face.  It  can  also  assess  users’  emotive  reactions  to  instruments.  Identifying  sources  of 
burden  and  reducing  the  causes  of  user  stress  result  in  a  more  efficient  Web-based  system. 

This  technical  report  summarizes  the  findings  of  a  data  collection  effort  designed  to  assist 
the  CNPC  Task  Force  EXCEL  HPFD  project  manager  in  collecting  and  analyzing  usability  data, 
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identifying  relevant  user  issues,  and  developing  recommendations  as  they  relate  to  automated 
human  performance  measurement  and  management  of  the  Navy’s  human  capital. 
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2  Usability  Testing  Literature  Review 


A  succinct  definition  of  usability  testing  is  found  in  Dumas  and  Redish’s  (1993)  popular 
handbook,  A  Practical  Guide  to  Usability  Testing.  The  authors  note  that  usability  testing’s 
primary  goal  is  to  improve  the  usability  of  the  product.  Specific  goals  and  concerns  are 
articulated  when  planning  each  test.  For  example,  for  the  usability  testing  of  the  Navy’s  Web- 
based  performance  management  tool,  a  specific  goal  was  to  assess  the  different  usability  needs 
for  supervisors  and  non-supervisors,  as  well  as  for  shipboard  and  non-shipboard  Sailors.  In  a 
usability  test,  the  following  four  key  factors  must  be  present: 

•  The  participants  represent  real  users. 

•  The  participants  do  real  tasks. 

•  The  usability  researcher  observes  and  records  what  participants  do  and  say. 

•  The  usability  researcher  analyzes  the  data,  diagnoses  the  problems,  and  recommends 
changes  to  fix  the  problems  (Dumas  &  Redish,  1999). 

Nielsen  (1993,  p.  165)  describes  usability  testing  as  “the  most  fundamental  usability  method” 
and  “irreplaceable”  because  it’s  the  only  mechanism  that  allows  the  researcher  to  obtain  direct, 
detailed  information  on  the  users’  experience  with  the  product  or  tool  being  tested 

Usability  researchers  agree  that  multiple  methodologies  can  be  used  to  effectively  assess 
the  user  experience.  In  fact,  most  usability  test  plans  include  several  types  of  data  collection. 
Methods  used  include  baseline  tests  of  existing  products  to  assess  pre-existing  problems,  surveys 
of  user  needs,  focus  groups  with  users,  participatory  design  experiences,  heuristic  evaluations, 
task  analysis,  and  paper  prototyping.  Among  all  the  usability  assessment  practices  that  are 


4 


employed,  the  two  most  consistently  emphasized  are  an  iterative  design  and  the  consideration  of 
the  user  context. 

2.1  Iterative  Design 

In  a  survey  of  usability  researchers,  Nielsen  (1993)  identified  the  top  six  methods  for 
usability  improvement.  Iterative  design  (tied  with  task  analysis)  was  number  one  on  his  list. 
There  are  several  reasons  why  iterative  design  of  usability  tests  is  so  important.  Changes  to  a 
system  as  a  result  of  usability  testing  sometimes  do  not  solve  the  problem.  In  fact,  new  solutions 
may  create  new  problems.  Furthermore,  new  solutions  may  reveal  additional  problems  that  were 
previously  hidden  or  outbalanced  by  the  original  problem  identified.  Nielsen’s  research 
analyzing  the  effectiveness  of  iterative  testing  found  a  median  improvement  in  system  usability, 
defined  by  the  various  usability  metrics  used  for  the  particular  test  plan,  of  38  percent  per 
iteration.  While  five  out  of  12  iterations  in  Nielsen’s  analysis  showed  that  one  dimension  of 
usability  had  gotten  worse,  significant  improvements  in  usability  continued  to  be  made  in  later 
iterations  (i.e.,  iterations  10,  11,  and  12). 

In  the  early  days  of  usability  testing  (the  1970s  and  1980s),  the  norm  was  one  large-scale 
test  of  30  users,  conducted  very  late  in  the  design  process  when  most  of  the  design  features  were 
stabilized  (and  thus  averse  to  change).  The  problem  with  this  approach  was  that  it  found 
pervasive  system  problems,  but  at  a  stage  in  the  development  cycle  where  it  was  too  late  to  fix 
them.  In  addition,  30  users  were  not  needed  to  identify  such  large  and  pervasive  problems.  The 
solution  adopted  was  to  test  earlier  prototypes  of  systems,  even  using  paper  prototypes  when 
necessary,  with  multiple  iterations  of  five  to  10  users.  This  approach  allows  the  identification  of 
large-scale  systemic  problems  early  on.  Since  1990,  iterative  testing  with  small  samples  has  been 
the  preferred  approach  (Dolan  &  Dumas,  1999). 
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2.2  Context  Awareness 


Valid  usability  measurement  cannot  take  place  outside  the  user’s  context,  and  usable 
systems  require  incorporating  this  context  into  the  development  cycle.  When  considering  tools 
such  as  guidelines  and  checklists  for  user-centered  design,  Bevan  and  Macleod  (1994)  warn 
against  dependence  on  checklists,  because  guidelines  for  usable  system  features  need  extensive 
detail  to  be  useful,  but  if  they  are  detailed  enough,  they  are  likely  to  be  too  specific  to  apply  in 
multiple  real-world  contexts.  For  example,  a  highly  interactive  Web-based  performance 
management  evaluation  form  that  requires  frequent  communication  with  a  server  to  complete 
may  be  desirable  in  an  office  setting  because  it  will  allow  the  user’s  data  to  be  saved  through 
many  interruptions.  Conversely,  this  may  not  be  desirable  on  board  a  deployed  Navy  ship,  since 
the  satellite  Internet  connection  may  be  unavailable  or  regularly  interrupted.  The  solution  is  to 
conduct  assessments  with  real-life  users  in  a  true-to-life  environment.  A  true-to-life  environment 
can  be  replicated  in  a  lab  setting,  but  the  most  realistic  setting  is  in  the  field  itself,  by  conducting 
on-site  usability  testing.  Bevan  and  Macleod  add  a  fifth  factor  to  Dumas  and  Redish’s  list  above: 
The  participant’s  real-life  context  is  represented  in  the  usability  test. 

This  evaluation  of  the  Navy’s  Web-based  performance  management  system  incorporated 
these  two  key  design  features  of  iterative  testing  and  context  awareness.  Usability  tests  were 
conducted  at  three  very  different  Navy  installations  with  time  between  iterations  to  make 
changes  to  the  system. 
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3  Study  Objectives 


The  objectives  for  this  study  were  to  capture  both  quantitative  and  objective  data  as  well 
as  qualitative  and  subjective  data  from  participants  to  identify  potential  sources  of  error  and  user 
burden.  Specifically,  the  objectives  of  this  study  were  to 

•  Conduct  usability  tests  on  the  HPFD  system  with  non-supervisory  Navy  personnel 
collecting  data  on  the  type  and  frequency  of  user  errors,  user  reactions  to  the  system, 
and  self-reported  user  satisfaction  with  the  system; 

•  Conduct  usability  tests  on  the  HPFD  and  ePerformance  systems  with  supervisory 
Navy  personnel  collecting  data  on  the  type  and  frequency  of  user  errors,  user 
reactions  to  the  systems,  and  self-reported  user  satisfaction  with  the  systems;  and 

•  Conduct  focus  group  interviews  among  non-supervisory  and  supervisory  personnel 
who  completed  HPFD  and  ePerformance  usability  tests  to  identify  features  that  users 
liked  or  features  that  need  to  be  improved. 

All  research  instruments  and  procedures,  including  participant  informed  consent  forms  for  both 
the  usability  testing  and  focus  group  interviews  were  reviewed  and  approved  by  the  research 
team’s  Institutional  Review  Board  (IRB).  Participants  were  briefed  on  the  purpose  of  the  study 
and  were  asked  to  read  and  sign  the  informed  consent  form  and  to  return  the  form  to  their 
respective  task  leaders.  No  adverse  events  occurred  during  the  course  of  this  study. 
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4  Participants 


A  local,  on-site  liaison  was  identified  by  the  project  manager  to  assist  in  participant 
recruiting,  scheduling,  and  study  logistics.  Instructions  were  sent  to  the  on-site  liaison  that 
described  the  criteria  for  selecting  potential  participants — supervisory  and  non-supervisory 
personnel  assigned  to  operational  and  shore  commands  or  units,  ranging  in  paygrade  from  E2 
through  06. 

4.1  Iteration  1 :  Naval  Air  Station  (NAS)  Brunswick 

Iteration  1  took  place  at  NAS  Brunswick  in  Brunswick,  Maine,  from  June  21,  2004, 
through  June  25,  2004.  A  total  of  21  active  duty  Navy  personnel  took  part  in  data  collection.  All 
21  personnel  participated  in  the  usability  testing  and  completed  the  pre-  and  post-test  usability 
surveys.  Of  the  21  personnel,  14  were  supervisors,  and  seven  were  in  non-supervisory  roles.  Ten 
participants  were  NAS  Brunswick  personnel,  ten  were  squadron  personnel,  and  one  participant 
was  from  a  ship  pre-commissioning  unit.  Only  one  of  these  personnel  could  not  participate  in  the 
subsequent  focus  group  interview. 

4.2  Iteration  2:  USS  KITTY  HAWK  (CV63) 

Iteration  2  took  place  aboard  the  USS  KITTY  HAWK  (CV63)  in  Yokosuka,  Japan,  from 
July  12,  2004,  through  July  16,  2004.  A  total  of  20  active  duty  Navy  personnel  were  scheduled  to 
part  in  data  collection.  Seventeen  personnel  participated  in  the  usability  testing  and  completed 
the  pre-  and  post-test  usability  surveys.  One  participant  could  not  be  tested  because  the  online 
system  was  unavailable,  and  another  participant  could  not  be  tested  because  the  ship’s  T1  line 
was  disconnected  to  switch  over  to  a  satellite  Internet  connection.  A  third  participant  could  not 
make  the  usability  session  because  shipboard  duties  created  a  scheduling  conflict.  Of  the  20 
personnel,  14  were  supervisors,  and  six  were  in  non-supervisory  roles.  Seven  were  officers,  and 
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13  were  enlisted.  Participants  were  from  the  following  departments:  six  from  the  Air 
Department,  five  from  Air  Intermediate  Maintenance  Department  (AIMD),  four  from  the 
Operations  Department,  two  from  the  Combat  Systems  Department,  one  from  Executive  Officer 
Administration,  one  from  Supply,  and  one  from  Weapons. 

Sixteen  of  the  usability  participants  took  part  in  focus  group  interviews,  and  four  were 
unavailable  for  the  focus  group  session.  Two  Navy  personnel  recruited  for  the  usability  portion 
of  the  study  who  were  not  able  to  complete  at  least  some  portion  of  the  usability  test  were 
excused  from  focus  group  interviews.  Two  additional  personnel  did  not  participate  in  focus 
group  interviews  because  shipboard  duties  created  a  scheduling  conflict. 

4.3  Iteration  3:  Kitsap  Naval  Base  -  Bangor 

Iteration  3  took  place  in  the  Trident  Training  Facility  at  Kitsap  Naval  Base  in  Bangor, 
Washington,  from  August  9,  2004,  through  August  13,  2004.  A  total  of  20  active  duty  Navy 
personnel  were  scheduled  to  take  part  in  data  collection.  Nineteen  personnel  participated  in  the 
usability  testing  and  completed  the  pre-  and  post-test  usability  surveys.  Of  the  19  personnel,  10 
were  supervisors,  and  nine  were  in  non-supervisory  roles.  All  Navy  personnel  were  enlisted 
Sailors.  Participants  were  from  the  following  commands:  six  from  the  USS  ALABAMA  (SSBN 
731),  six  from  the  USS  ALASKA  (SSBN  732),  one  from  the  USS  NEVADA  (SSBN  733),  four 
from  the  USS  KENTUCKY  (SSBN737),  one  from  Commander  Submarine  Squadron  Nineteen 
(CSS-19),  and  one  from  Commander  Submarine  Squadron  Seventeen  (CSS-17). 

Thirteen  usability  participants  took  part  in  focus  group  interviews,  and  seven  participants 
did  not  participate  in  focus  group  interviews  due  to  scheduling  conflicts.  Of  the  13  participating 
personnel,  six  were  supervisors,  and  seven  were  in  non-supervisory  roles.  All  Navy  personnel 
were  enlisted  Sailors.  Participants  were  from  the  following  commands:  five  from  the  USS 
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ALABAMA  (SSBN  731),  four  from  the  USS  ALASKA  (SSBN  732),  one  from  the  USS 
NEVADA  (SSBN  733),  one  from  the  USS  KENTUCKY  (SSBN737),  one  from  Commander 
Submarine  Squadron  Nineteen  (CSS- 19),  and  one  from  Commander  Submarine  Squadron 
Seventeen  (CSS- 17). 
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5  Instruments  and  Procedures 


5.1  Usability  scenarios 

Usability  scenarios  were  developed  to  evaluate  the  effectiveness  of  screen  layouts, 
performance  appraisal  item  structures,  and  on-screen  features  for  the  Navy’s  HPFD  and 
ePerformance  systems.  Specifically,  usability  testing  protocol  and  scenarios  targeted  the 
following  potential  problems: 

•  Unclear  navigational  instructions.  Are  respondents  able  to  tell  where  on  the  screen  to 
start  reading  and  where  to  supply  the  required  information? 

•  Confusing  help  text.  Is  help  text  consistently  displayed  within  the  documents,  and 
does  the  help  text  answer  the  users’  most  common  questions? 

•  Meaningless  or  off-putting  error  messages.  Are  error  messages  appropriately 
displayed  when  problems  occur?  Do  respondents  find  the  error  messages  informative 
and  helpful  rather  than  alarming  and  off-putting? 

•  Problems  of  accessing/responding  via  the  Web.  What  is  the  most  efficient  Web  tool 
design  for  the  least  capable  information  technology  (IT)  platform  and  least  advanced 
hardware  and  software? 

Additionally,  test  scenarios  were  developed  to  simulate  actual  tasks  that  Navy  non-supervisors 
and  supervisors  are  likely  to  encounter. 

In  an  effort  to  test  the  HPFD  and  ePerformance  systems  in  the  field,  this  research  study 
utilized  a  portable  usability  lab — a  coordinated  system  of  digital  audio  and  video  data  capture 
equipment.  The  portable  usability  lab  features  professional  grade  video  monitoring  and  recording 
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capabilities  including  two  high-resolution  video  cameras  with  silent  remote  control  pan,  tilt, 
zoom,  and  focus. 

Following  the  best  practices  in  usability  testing  described  above,  an  iterative  approach 
with  three  separate  rounds  of  usability  testing  and  focus  group  interviews  was  used.  In  order  to 
obtain  the  perspectives  and  assess  the  experiences  of  the  diverse  Navy  workforce,  it  was 
important  to  include  participants  from  a  variety  of  work  environments  in  a  variety  of  geographic 
locations.  As  a  result,  our  research  plan  included  usability  testing  among  Sailors  in  a  variety  of 
warfare  communities  (i.e.,  surface,  submarine,  and  aviation  communities)  in  an  Atlantic  Fleet 
(i.e.,  NAS  Brunswick),  Pacific  Fleet  (Naval  Base  Kitsap  -  Bangor),  and  overseas  (USS  KITTY 
HAWK  [CV63] — Yokosuka,  Japan)  locations. 

5.2  Usability  survey 

Two  paper-and-pencil  self-administered  surveys — pre-test  and  post-test  surveys — were 
developed  to  obtain  Navy  personnel’s  subjective  impressions  of  the  HPFD  and  ePerformance 
systems.  The  objective  of  the  participant  survey  was  to  obtain  data  on  users’  subjective  reactions 
to  the  Web-based  tool  and  assess  ease  of  use,  professional  value,  personal  value,  and  overall 
satisfaction  with  the  Navy’s  new  performance  appraisal/management  tool. 

The  pre-test  survey  included  items  related  to  participant  demographics  (e.g.,  age,  gender, 
race/ethnicity,  education,  pay  grade,  and  time  on  active  duty),  frequency  of  computer  use  both  at 
home  and  at  work,  prior  experience  with  PeopleSoft  software,  satisfaction  with  the  current 
performance  appraisal  process,  satisfaction  with  advancement/promotion  process,  and  perceived 
difficulty  with  the  HPFD  and  ePerformance  systems  prior  to  use.  Items  assessing  satisfaction 
with  the  current  performance  appraisal  process  and  satisfaction  with  the  advancement/promotion 
process  were  adapted  from  the  2000  Navy-wide  Personnel  Survey  (Olmsted  &  Underhill,  2003). 
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The  post-test  survey  asked  participants  to  report  several  aspects  related  to  completing  the 
tasks  in  the  usability  portion  of  this  study.  Specifically,  items  asked  about  perceived  comfort 
completing  the  tasks,  perceptions  about  how  successful  participants  were  in  completing  the  tasks, 
ease  of  use  compared  to  other  systems,  overall  perceived  ease  of  use,  how  difficult  the  system 
was  to  understand,  perceived  appearance  of  the  system,  perceived  efficiency  of  the  system, 
acclimation  or  gradual  improvement  of  use  while  using  the  system,  satisfaction  with  the  current 
performance  appraisal  process,  satisfaction  with  the  advancement/promotion  process,  and  overall 
satisfaction  with  the  pilot  HPFD  and  ePerformance  systems. 

5.3  Focus  group  guide  and  questions 

Focus  group  interviews  were  conducted  as  a  means  of  identifying  features  of  the  HPFD 
or  ePerformance  systems  that  users  liked  or  features  that  needed  to  be  improved.  Following  the 
conventions  and  best  practices  of  conducting  focus  group  interviews  (Edmunds,  1999;  Morgan, 
1997;  Krueger,  2003),  homogeneous  groups  of  six  to  10  participants  were  scheduled  and 
interviewed  in  a  private  on-site  room.  Groups  consisted  of  personnel  with  similar  performance 
appraisal/management  responsibilities  and  work  environments  (e.g.,  supervisory/non- 
supervisory,  ship/shore). 

The  focus  group  interview  questions  were  developed  to  capture  qualitative  information 
from  usability  test  participants.  Questions  centered  on  aspects  of  the  usability  testing  process — 
logging  onto  the  system,  opening  the  test  document,  using  the  system  (HPFD,  ePerformance,  or 
both  HPFD  and  ePerformance),  and  closing  the  document.  A  final  question  asked  for  a  summary 
rating  of  the  system  participants  used,  reasons  for  that  rating,  and  what  might  be  done  to  improve 
their  rating. 
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Focus  group  interviews  were  conducted  with  Navy  supervisors  and  non- supervisors  in 
three  iterations  at  three  separate  base  locations.  Initially,  focus  group  interviews  were  scheduled 
to  consist  of  heterogeneous  group  members — supervisory  personnel  and  non-supervisory 
personnel — since  supervisory  personnel  completed  both  an  HPFD  and  ePerformance  document, 
and  non-supervisory  personnel  completed  only  the  HPFD  document.  Occasionally,  due  to 
scheduling  challenges,  the  base  recruiting  liaison  was  not  able  to  schedule  heterogeneous  groups. 
Additionally,  due  to  hardware  and  software  problems,  some  usability  testing  participants  were 
limited  in  the  test  applications  they  were  able  to  complete.  As  a  result,  some  questions  pertained 
to  only  a  portion  of  participants  in  several  groups,  but  data  were  collected  from  participants  for 
those  portions  of  the  usability  test  that  they  completed. 
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6  Results 


6.1  Survey  Data 

Demographic  data  from  the  user  surveys  appear  in  Table  1.  Results  are  presented  overall 
and  are  also  divided  among  supervisory  and  non- supervisory  personnel.  Study  participants  were 
between  the  ages  of  18  and  44,  with  the  non-supervisors  between  18  and  34  years  of  age  and  the 
supervisors  between  the  ages  of  25  and  44.  Most  participants  (about  77  percent)  were  enlisted 
Sailors  in  the  paygrades  of  E2-E9.  Thirteen  participants  were  officers.  There  were  no  warrant 
officers  in  the  pre-test.  Eighty-nine  percent  of  the  participants  were  male,  with  only  slight 
variance  between  supervisors  and  non- supervisors.  Racial  backgrounds  were  only  slightly 
different  between  non-supervisors  and  supervisors,  as  well.  Also,  80  percent  of  supervisor 
participants  were  white,  and  68%  of  non-supervisor  participants  were  white. 

Not  surprisingly,  the  clearest  demographic  distinctions  between  supervisory  and  non- 
supervisory  participants  were  in  educational  background  and  years  in  the  Navy.  About  50 
percent  of  the  non-supervisors  had  only  a  high  school  diploma  or  an  equivalent  degree, 
compared  to  20  percent  of  the  supervisors.  A  majority  of  non-supervisors  participating  in  the 
tests  had  been  in  the  Navy  five  years  or  fewer.  Supervisors,  on  the  other  hand,  had  mostly  been 
in  the  Navy  for  10  years  or  more.  Overall,  the  demographic  data  show  no  major  surprises. 

In  addition  to  demographic  data,  the  pre-test  and  post-test  surveys  collected  other 
information  expected  to  relate  to  the  users’  perceptions  of  the  Web-based  performance 
management  system.  Questions  addressed  topics  such  as  the  participants’  experiences  using 
computers  and  their  overall  impressions  of  the  test  version  of  Web-based  tool.  The  survey  results 
show  that  almost  all  Sailors  used  a  computer  frequently  both  at  home  and  at  work.  Table  2  shows 
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Table  1.  Demographic  Data  from  Pretest  Survey 


Overall 

Non-Supervisory 

Personnel 

Supervisory 

Personnel 

Percent 

N 

Percent 

N 

Percent 

N 

Age 

18-24 

21.05 

12 

45.45 

10 

5.71 

2 

25-34 

43.86 

25 

45.45 

10 

42.86 

15 

35-44 

31.58 

18 

9.09 

2 

45.71 

16 

45-54 

3.51 

2 

0 

0 

5.71 

2 

55+ 

0 

0 

0 

0 

0 

0 

Paygrade 

E2  -  E3 

12.28 

7 

31.82 

7 

0 

0 

E4  -  E6 

36.84 

21 

63.64 

14 

20 

7 

E7  -  E9 

28.07 

16 

4.55 

1 

42.86 

15 

W1  -  W4 

0 

0 

0 

0 

0 

0 

01  -03 

12.28 

7 

0 

0 

20 

7 

04-06 

10.53 

6 

0 

0 

17.14 

6 

Gender 

Male 

89.54 

51 

90.91 

20 

Female 

10.46 

6 

9.09 

2 

n 

Ethnic  Origin1 

American  Indian  or  Alaska  Native 

5.26 

3 

9.09 

2 

2.86 

i 

Asian  (e.g.,  Asian  Indian,  Chinese,  Filipino, 
Japanese,  Korean,  Vietnamese,  etc.) 

1.75 

1 

0 

0 

2.86 

i 

Black  or  African-American 

10.53 

6 

18.18 

4 

5.71 

2 

Native  Hawaiian  or  other  Pacific  Islander 
(e.g.,  Samoan,  Guamanian,  Chamorro,  etc.) 

0 

0 

0 

0 

0 

0 

White 

75.44 

43 

68.18 

15 

80 

28 

Other 

8.77 

5 

18.18 

4 

2.86 

1 

Hispanic  Ethnicity 

Hispanic 

7.02 

4 

13.64 

3 

2.86 

1 

Not  Hispanic 

92.98 

53 

86.36 

19 

97.14 

34 

Educational  Background 

Less  than  high  school  completed/no  diploma 

0 

0 

0 

0 

0 

0 

Alternate  degree,  GED,  home  study,  or 
adult-school  certification 

5.36 

3 

9.09 

2 

2.86 

1 

High  school  graduate/diploma 

23.63 

15 

40.91 

9 

17.14 

6 

Some  college,  no  degree 

33.33 

19 

45.45 

10 

31.43 

11 

Associate’s  degree  or  other  two-year  degree 

1.53 

6 

4.55 

1 

14.29 

5 

(A. A.,  A.S.,  etc.) 

Bachelor’s  degree  (B.A.,  B.S.,  etc.) 

21.05 

12 

0 

0 

34.29 

12 

Master’s  degree  (M.A.,  M.S.,  M.B.A.,  etc.) 

1.75 

1 

0 

0 

2.86 

1 

Doctoral  or  professional  degree  (J.D.,  Ph.D., 
D.Ph.,  M.D.,  etc.) 

1.75 

1 

0 

0 

2.86 

1 

Number  of  Years  in  the  Navy 

Less  than  1  Year 

3.51 

2 

9.09 

2 

0 

0 

1-5  Years 

31.58 

18 

59.09 

13 

14.29 

5 

5-10  Years 

8.77 

5 

9.09 

2 

8.57 

3 

10-15  Years 

28.07 

16 

18.18 

4 

34.29 

12 

More  than  15  Years 

28.07 

16 

4.55 

1 

42.86 

15 

1  Note  that  the  total  count  for  this  variable  is  above  57,  the  number  of  participants  who  completed  the  usability 
survey.  Ethnic  origin  is  a  “mark  all  that  apply”  question  and  some  users  selected  multiple  answers. 
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that  about  85  percent  of  participants  said  that  they  either  used  a  computer  at  work  every  day,  or 


that  most  of  their  work  was  done  on  a  computer.  Furthermore,  at  home,  about  37  percent  used  a 
computer  every  day,  and  another  58  percent  used  a  computer  sometimes.  Discrepancies  exist 
between  supervisors  and  non-supervisors.  About  32  percent  of  non-supervisors  reported  using  a 
computer  at  work  only  sometimes  or  never,  compared  to  only  three  percent  of  supervisors.  Nine 
percent  of  non-supervisors  never  used  a  computer  at  home,  compared  to  three  percent  of 
supervisors.  Based  on  these  demographics,  non-supervisors,  who  were  more  likely  to  be 
computer  novices,  might  be  expected  to  have  more  problems  using  the  Web-based  performance 
management  system. 

Table  2.  Computer  Use  at  Home  and  at  Work 


Overall 

Non-Supervisory 

Personnel 

Supervisory 

Personnel 

How  often  do  you  use  a  computer... 

Percent 

N 

Percent 

N 

Percent 

N 

...at  work? 

Never 

1 

4.55 

1 

0.00 

0 

Sometimes,  but  not  every  day 

27.27 

6 

2.86 

1 

Every  day,  but  not  all  day 

36.36 

8 

45.71 

16 

Most  of  my  work  is  on  a  computer 

43.86 

31.82 

7 

51.43 

18 

...at  home? 

Never 

5.26 

3 

9.09 

2 

2.86 

1 

Sometimes,  but  not  every  day 

57.89 

33 

50.00 

11 

62.86 

22 

1  use  a  computer  every  day. 

36.84 

21 

40.91 

9 

34.29 

12 

When  participants’  perceptions  of  overall  ease  of  using  the  Web-based  system  are 
considered,  supervisors  and  non-supervisors  appear  to  have  had  somewhat  similar  impressions, 
with  more  non-supervisors  having  difficulty  with  the  system.  Table  3  shows  their  responses  to 
the  question,  “Overall,  how  easy  or  difficult  was  the  system  to  use?”  About  three  percent  of 
supervisors  said  it  was  very  difficult  to  use,  and  about  12  percent  of  supervisors  said  it  was 
somewhat  difficult  to  use.  Among  non-supervisors,  none  reported  that  it  was  very  difficult  to 
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use,  but  33  percent  said  it  was  somewhat  difficult.  About  nine  percent  of  supervisors  and 


non-supervisors  reported  that  the  system  was  very  easy  to  use. 

Table  3.  Overall  Ease  of  Using  the  System 


Easy  or  difficult 
to  Use  the  System? 

Overall 

Supervisory 

Non-Supervisory 

Percent 

N 

Percent 

N 

Percent 

N 

Very  Difficult  to  Use 

1.82 

1 

2.94 

1 

0.00 

0 

Somewhat  Difficult  to  Use 

20.00 

11 

11.76 

4 

33.33 

7 

Neither  Difficult  nor  Easy  to  Use 

18.18 

10 

20.59 

7 

14.29 

3 

Somewhat  Easy  to  Use 

50.91 

28 

55.88 

19 

42.86 

9 

Very  Easy  to  Use 

9.09 

5 

8.82 

3 

9.52 

2 

In  addition  to  these  questions,  the  participants  answered  a  series  of  questions  about  their 
perceptions  of  the  fairness  and  accuracy  of  the  current  EVAL/FITREP  system  as  well  as  the  test 
version  of  the  Web-based  system.  The  overall  results  and  results  for  supervisors  and  non¬ 
supervisors  for  all  questions  in  both  the  pre-test  and  post-test  surveys  are  detailed  in  Appendix  A. 

6.2  Usability  Data 

In  this  section,  we  present  the  results  of  the  usability  testing  along  two  dimensions:  the 
time  it  took  to  complete  specific  tasks  and  the  number  and  types  of  problems  that  were  observed 
while  users  were  completing  specific  scenarios.  For  the  usability  testing,  we  developed  one  set  of 
scenarios  for  supervisors  and  a  second  set  of  scenarios  for  non-supervisors.  Supervisors  were 
asked  to  complete  the  HPFD  session  as  well  as  the  ePerformance  appraisal  document.  Non¬ 
supervisors  were  asked  to  complete  only  the  HPFD  session.  Table  4  presents  the  durations  for 
completing  each  task  for  non-supervisors  and  supervisors. 

Several  observations  are  apparent  when  examining  the  duration  data  presented  in 
Table  4.  First  of  all,  as  expected,  the  tutorial  task  took  the  most  time.  On  average,  it  took  non¬ 
supervisors  22  minutes  and  12  seconds,  and  it  took  supervisors  28  minutes  and  24  seconds.  The 
supervisors  took  longer  because  they  were  asked  to  complete  one  additional  section  in  the 
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Table  4.  Durations  for  Completing  Usability  Tasks 


Non-Supervisor 

Supervisor 

No. 

Task 

Avg 

Min 

Max 

Avg 

Min 

Max 

HPFD  Tasks 

1 

Complete  the  CBT  Tutorial. 

0:22:12 

0:02:46 

0:42:13 

0:28:24 

0:02:09 

0:44:25 

2 

Log  in  to  NSIPS. 

0:03:51 

0:01:07 

0:27:14 

0:01:52 

0:00:23 

0:20:32 

3 

Open  the  HPFD  document. 

0:02:40 

0:00:47 

0:09:55 

0:04:58 

0:00:42 

0:07:33 

4 

Complete  the  HPFD  document. 

0:12:22 

0:02:44 

0:25:41 

0:11:04 

0:02:55 

0:26:40 

5 

Check  spelling. 

0:01:23 

0:00:04 

0:02:55 

0:01:33 

0:00:05 

0:06:10 

6 

Find  the  “Target  Behaviors” 
description. 

0:01:22 

0:00:13 

0:02:22 

0:00:59 

0:00:11 

0:01:28 

7 

Change  ratings  and  cut  and  paste 
comments. 

0:01:36 

0:00:37 

0:03:19 

0:01:34 

0:00:24 

0:04:53 

8 

Collapse  all  sections  of  the 
document. 

0:00:47 

0:00:05 

0:01:43 

0:00:40 

0:00:10 

0:01:27 

9 

Submit  the  HPFD  document. 

0:00:43 

0:00:05 

0:01:43 

0:01:24 

0:00:11 

0:07:13 

10 

Enter  a  performance  note. 

0:02:28 

0:00:47 

0:06:12 

0:01:17 

0:00:37 

0:08:11 

ePerformance  Tasks 

11 

Log  out  of  PeopleSoft. 

- 

- 

- 

0:00:15 

0:00:11 

0:00:18 

12 

Log  into  NSIPS  using  the 
ePerformance  test  account  (System 
Administrator  account). 

- 

- 

- 

0:01:23 

0:00:44 

0:02:25 

13 

Open  the  Annual  Performance 
Appraisal  1  document. 

- 

- 

- 

0:04:22 

0:01:58 

0:08:10 

14 

Complete  the  Annual  Performance 
Appraisal  1  document. 

" 

" 

- 

0:10:34 

0:03:29 

0:16:02 

15 

Check  the  ratings  descriptions  for 
one  dimension. 

- 

- 

- 

0:00:45 

0:00:15 

0:01:06 

16 

Check  spelling. 

- 

- 

- 

0:00:47 

0:00:11 

0:02:05 

17 

Check  language. 

- 

- 

- 

0:00:39 

0:00:07 

0:01:09 

18 

Calculate  ratings. 

- 

- 

- 

0:00:42 

0:00:05 

0:03:46 

19 

Submit  the  Annual  Performance 
Appraisal  1  document. 

- 

- 

- 

0:05:06 

0:00:11 

0:12:53 
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tutorial,  describing  manager  functions.  Second,  other  than  the  tutorial,  there  are  no  discernible 
differences  between  the  durations  for  non-supervisors  and  for  supervisors.  In  general,  the 
average,  minimum,  and  maximum  times  for  completing  tasks  within  the  HPFD  session  are 
similar  across  the  two  groups.  Third,  the  table  shows  some  extremely  long  durations  for 
completing  some  of  the  tasks.  Some  examples  are  listed  below. 

•  It  took  27  minutes  and  14  seconds  for  one  non-supervisor  to  log  into  NSIPS. 

•  The  maximum  lengths  of  time  for  opening  the  HPFD  document  were  9  minutes  and 
55  seconds  for  non-supervisors  and  7  minutes  and  33  seconds  for  supervisors. 

•  One  supervisor’s  completed  HPFD  document  took  7  minutes  and  13  seconds  to 
submit. 

•  One  supervisor’s  completed  ePerformance  Appraisal  took  12  minutes  and  53  seconds 
to  submit. 

These  extremely  long  durations  for  completing  some  very  simple,  one-step  tasks  are  likely  the 
result  of  server  and  connectivity  problems.  Further  analysis  in  Technical  Report  #2  will  indicate 
whether  these  long  durations  were  observed  more  frequently  at  one  of  the  sites  (such  as  on  the 
USS  KITTY  HAWK  [CV63],  where  connectivity  seemed  to  be  the  slowest).  These  durations 
demonstrate  how  fairly  uncomplicated  problems  such  as  a  connectivity  interruption  can 
indiscriminately  affect  the  user’s  experience  in  completing  documents  in  NSIPS.  The  additional 
coding  of  usability  problems,  detailed  below,  will  provide  more  information  as  to  the  details  of 
users’  experience.  The  analysis  of  usability  errors  in  each  of  the  tasks  presents  a  different  picture 
of  the  users’  experiences  with  the  test  scenarios.  Whereas  the  timing  estimates  present  a  picture 
of  the  overall  burdens  users  faced,  such  as  extremely  long  durations  when  attempting  to  log  into 
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NSIPS,  the  usability  errors  allow  an  understanding  of  specifically  what  kinds  of  problems  caused 
those  burdens. 

Table  5  displays  the  count  of  problem  incidents  for  Task  1,  completing  the  CBT  tutorial 


on  HPFD.  In  the  action  memos  submitted  after  each  data  collection  iteration,  it  was  noted  that 


while  users  completing  the  tutorial  appeared  to  read  most  of  the  text  on  the  screens,  they  did  not 
generally  complete  the  interactive  portions  of  the  tutorial.  On  many  screens,  it  appeared  difficult 
for  users  to  see  where  they  were  being  asked  to  enter  data  or  click  buttons  within  the  tutorial 
screens.  This  observation  is  supported  by  the  usability  errors  analysis.  The  greatest  problem 
observed  was  that  users  did  not  follow  screen  instructions.  This  happened  125  times  for  non¬ 
supervisors  and  170  times  for  supervisors.  When  it  happed  for  non-supervisors,  it  happened 
about  18  times  per  session,  and  when  it  happened  for  supervisors,  it  happened  about  24  times  per 
session.  This  error  incident  rate  is  very  high.  None  of  the  other  tasks  completed  in  all  of  the 
usability  testing  featured  this  many  problems.  The  best  likely  explanation  for  this  is  that  the 


tutorial  was  a  long  task. 

Table  5.  Task  1:  Complete  the  CBT  Tutorial 


Problem  Category 

Non-Supervisors 

Supervisors  i 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  does  not  follow  screen  instructions 

125 

17.8 

170 

24.2 

Tutorial  button  error 

16 

5.33 

42 

6 

User  asks  for  help 

3 

1.5 

1 

1 

Navigational  error 

1 

1 

0 

0 

As  displayed  in  Table  4  above,  it  took  non- supervisors  an  average  of  22  minutes  and 


supervisors  an  average  of  28  minutes  to  complete  the  tutorial.  The  longer  the  task,  the  more 


opportunity  a  user  has  for  error.  This  observation  likely  explains  the  higher  rate  of  problem 
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incidents  for  supervisors.  Supervisors  experienced  a  higher  number  of  total  problems  and  a 


higher  average  number  of  incidents  per  session  because  they  spent  an  average  of  6  minutes 
longer  completing  the  “Manager  Functions”  section  of  the  tutorial.  In  general,  Table  5  shows 
that  the  other  problems  that  occurred  during  the  tutorial  were  fairly  minor.  Users  did  make  errors 
clicking  on  the  buttons  unique  to  the  tutorial.  The  lack  of  clarity  as  to  which  parts  of  the  tutorial 
were  interactive  and  which  were  not  likely  contributed  to  these  errors  as  well. 

For  Task  2,  users  were  asked  to  log  into  NSIPS  using  their  new  account,  created  for  the 
purpose  of  the  usability  testing.  Table  6  displays  the  problem  categories  and  incidents  for  Task  2. 
The  primary  problem  observed  with  this  task  was  setting  the  new  NSIPS  password.  Upon 
logging  in  the  first  time,  users  were  required  to  change  their  password  to  meet  specific  criteria  of 
eight  characters  with  at  least  two  numbers  and  at  least  one  capitalized  letter.  Users  were  not 
given  instructions  on  the  password  requirements  until  after  they  tried  a  specific  password  and 
failed.  Across  all  sessions,  problems  setting  the  new  password  occurred  a  total  of  37  times.  It 


happened  more  frequently  (21  times  vs.  16  times)  for  non-supervisors  than  for  supervisors.  Note 

Table  6.  Task  2:  Log  into  NSIPS 


Non-Supervisors 

Supervisors  j 

Problem  Category 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  not  able  to  set  new  password. 

21 

3 

16 

2.28 

User  asks  for  help 

7 

1.16 

11 

1.57 

System  or  server  error 

7 

2.33 

7 

1.4 

User  refers  to  QRG 

3 

3 

1 

1 

Navigational  error 

2 

2 

0 

0 

User  is  timed  out 

1 

1 

0 

0 

User  retries  action  because  the  system 
did  not  react  the  first  time. 

1 

1 

0 

0 
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that  when  this  problem  occurred,  it  occurred  about  three  times  per  session  for  non-supervisors 
and  about  two  times  per  session  for  supervisors. 

Additional  problems  were  observed  with  logging  into  NSIPS.  Users  asked  for  help 
relatively  frequently  (compared  to  other  tasks).  Non- supervisors  asked  for  help  seven  times,  and 
supervisors  asked  for  help  eleven  times.  Two  other  problems  occurring  with  this  task  were 
similar:  the  system  or  server  error  and  the  timing  out  error.  The  system  or  server  error  happened 
seven  times  each  for  non-supervisors  and  supervisors.  This  error  was  recognizable  by  extreme 
slowness  in  the  connection  speed.  The  system  or  server  error  problem  was  coded  when  the 
screen  did  not  respond  or  continually  showed  a  partially  complete  loading  bar  at  the  bottom  of 
the  screen.  The  timing  out  error,  on  the  other  hand,  happened  only  once  for  Task  2.  Timing  out 
was  observed  when  the  user  was  returned  to  the  login  screen.  This  error  was  observed  more 
frequently  in  the  HPFD  document  when  the  user  would  click  on  a  button  that  led  to  a  screen 
saying,  “This  page/document  is  currently  unavailable.”  In  general,  the  system  or  server  error  is 
attributable  to  slow  connection  speed  or  server  down  time,  whereas  the  timing  out  error  is 
attributable  to  the  PeopleSoft  settings. 

Several  other  problems  or  behaviors  were  observed  at  a  minor  level  when  completing 
Task  2.  These  included  use  of  the  QRG,  navigational  errors,  and  retrying  an  action  because  the 
system  failed  to  react  the  first  time.  Users  tended  to  retry  actions  when  they  faced  a  system  or 
server  problem.  Overall,  more  problems  were  observed  among  non-supervisors  than  supervisors. 

Task  3,  opening  the  HPFD  document,  was  the  first  task  users  were  asked  to  complete 
within  the  PeopleSoft  performance  management  system.  The  most  significant  problems  users 
experienced  were  navigational  errors  while  attempting  to  find  the  location  of  the  HPFD 
document.  Participants  had  trouble  identifying  the  correct  path  (Employee  Self-Service  - 
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Performance  Management  -  Performance  Documents)  from  the  range  of  options  available  to 


them  in  the  left-side  menu.  Table  7  shows  that  this  happened  more  for  non- supervisors  (25 
times)  than  for  supervisors  (19  times).  Also  noteworthy  in  Table  7  is  the  extensive  use  of  the 
QRG.  Both  non-supervisors  and  supervisors  used  it,  at  a  rate  of  21  times  and  14  times, 
respectively.  On  average,  non- supervisors  who  consulted  the  QRG  used  it  three  times. 
Supervisors  who  consulted  the  QRG  used  it  two  times  on  average.  In  addition  to  using  the  QRG, 


test  participants  also  asked  for  help. 

Table  7.  Task  3:  Open  the  HPFD  document 


Non-Supervisors 

I  Supervisors  1 

Problem  Category 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Navigational  error 

25 

3.57 

19 

2.71 

User  refers  to  QRG 

21 

3 

14 

2 

User  asks  for  help 

8 

1.14 

7 

1.4 

System  or  server  error 

6 

1 

7 

1.16 

User  retries  action  because  the  system 
did  not  react  the  first  time. 

5 

1.66 

1 

1 

User  is  timed  out 

0 

0 

2 

1 

User  searches  outside  HPFD  session 

1 

1 

1 

1 

General  button  error 

1 

1 

0 

0 

Other  problems  occurred  less  frequently  when  users  attempted  to  open  the  HPFD 
document.  These  included  system  or  server  errors,  retrying  actions  due  to  inactivity,  searching 
for  documents  outside  the  HPFD  session  (such  as  in  the  Microsoft  Internet  Explorer  menus  or  on 
the  Internet),  and  a  general  button  error. 

After  opening  the  HPFD  document  in  Task  3,  users  were  required  by  Task  4  to  complete 
the  HPFD  form.  This  task  focused  solely  on  entering  ratings  and  comments.  Subsequent  tasks 
required  users  to  use  specific  functions  in  the  HPFD  document,  including  the  spelling  check  and 
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the  collapse  all  sections  features.  Table  8  shows  that  the  most  common  problem  situations  were 


the  use  of  the  QRG  and  the  user  asking  for  help.  Consistent  with  the  previous  two  tasks,  both  of 


these  problems  occurred  more  frequently  for  non- supervisors  than  supervisors.  The  QRG  was 


used  22  times  among  non-supervisors  but  only  six  times  among  supervisors,  with  non¬ 


supervisors  asking  for  help  eleven  times. 

Table  8.  Task  4:  Complete  the  HPFD  document 


Non-Supervisors 

|  Supervisors  | 

Problem  Category 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  refers  to  QRG 

22 

3.14 

6 

2 

User  asks  for  help 

11 

1.83 

9 

1.8 

General  button  error 

6 

1.2 

2 

1 

User  is  timed  out 

4 

1 

5 

2.5 

Navigational  error 

7 

2.33 

1 

1 

System  or  server  error 

2 

1 

0 

0 

User  retries  action  because  the  system 
did  not  react  the  first  time. 

3 

1.5 

0 

0 

User  searches  outside  HPFD  session 

2 

2 

1 

1 

Completing  the  HPFD  document  was  typically  also  a  longer  task.  Table  4  above  shows 
that  it  took  non-supervisors  about  12  minutes  and  supervisors  about  11  minutes  on  average  to 
complete.  This  longer  duration  likely  contributes  to  the  greater  diversity  of  problems  with 
relatively  high  incidence  rates  observed  for  Task  4.  Users  also  exhibited  general  button  errors 
and  navigational  errors  and  had  timing  out  problems  and  system  or  server  problems.  In  a  few 
instances,  users  attempted  to  retry  failed  actions  and  searched  outside  the  HPFD  session. 

The  next  five  tasks  (Tasks  5-9)  required  the  users  to  complete  very  specific  brief  tasks 
within  the  HPFD  document,  including  checking  the  spelling,  finding  the  target  behaviors 
description,  changing  ratings  and  comments,  collapsing  all  sections  of  the  HPFD  document,  and 
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submitting  the  HPFD  document.  The  details  of  the  problem  occurrences  for  these  tasks  can  be 
found  in  Appendix  B.  but  in  general,  fewer  incidents  occurred.  The  most  frequent  and  significant 
problem  was  timing  out.  Users  experienced  timing  out  problems  with  all  five  of  these  tasks,  and 
for  all  but  one  of  them,  it  was  the  most  frequent  problem  faced,  according  to  the  counts  from  the 
usability  data.  In  most  cases,  the  user  was  timed  out  of  the  HPFD  session  while  they  were 
working  in  Task  4,  completing  the  HPFD  document.  Then  when  they  attempted  to  do  the 
remaining  tasks  within  the  document,  they  were  unable  to  complete  them  due  to  the  timing  out 
problem.  It  is  noteworthy  that  in  most  cases,  the  HPFD  document  completion  times  for  users  did 
not  appear  to  be  extraordinarily  long.  In  fact,  since  they  were  aware  they  were  in  a  test 
environment,  most  users  felt  comfortable  taking  shortcuts  such  as  copying  and  pasting  text,  or 
writing  informal  appraisals  of  their  performance.  We  expect  that  the  actual  completion  times  for 
the  HPFD  session  may  be  even  longer  when  it  is  used  in  real  performance  evaluations. 

Task  10,  entering  a  performance  note,  was  the  final  task  in  the  HPFD  session.  This  task 
represented  only  the  second  time  users  were  asked  to  search  within  PeopleSoft  for  a  particular 
document.  As  in  Task  3,  the  predominant  problems  are  navigational  errors  and  the  need  to  refer 
to  the  QRG.  Table  9  displays  the  results  in  detail.  Among  non-supervisors,  there  were  16 
navigational  errors,  and  among  supervisors  there  were  nine  navigational  errors  when  attempting 
to  enter  a  performance  note.  Interestingly,  while  non-supervisors  used  the  QRG  a  relatively  high 
12  times,  one  supervisor  only  used  it  one  time.  The  other  categories  featured  fairly  minimal 
levels  of  problem  incidents. 

The  first  10  tasks,  described  above,  related  to  the  HPFD  session  and  were  completed  by 
both  supervisors  and  non-supervisors.  The  remaining  eight  tasks  dealt  with  completing  the 
ePerformance  Appraisal  document,  and  as  such  were  only  completed  by  supervisors.  Overall,  it 
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Table  9.  Task  10:  Enter  a  performance  note 


Non-Supervisors 

Supervisors  | 

Problem  Category 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number 
of  Incidents 
Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Navigational  error 

16 

2.28 

9 

2.25 

User  refers  to  QRG 

12 

1.71 

1 

1 

User  is  timed  out 

4 

1 

2 

1 

User  asks  for  help 

3 

1 

1 

1 

User  retries  action  because  the  system 
did  not  react  the  first  time. 

3 

1.5 

9 

4.5 

General  button  error 

0 

0 

2 

1 

System  or  server  error 

3 

1 

0 

0 

is  noteworthy  among  these  tasks  that  there  were  much  lower  rates  of  problem  incidences.  Most 
tasks,  including  logging  in  to  the  NSIPS  system  with  the  test  account,  completing  the  Annual 
Performance  Appraisal  1  document,  checking  ratings,  spelling,  and  language,  calculating  ratings, 
and  submitting  the  Performance  Appraisal  1  document,  had  five  or  fewer  incidents  for  even  the 
most  severe  problem.  This  may  be  evidence  of  a  learning  effect  after  first  completing  the  HPFD 
document.  However,  many  supervisors  did  not  complete  the  HPFD  document  and  started  with 
the  ePerformance  Appraisal. 

Only  one  task  among  the  ePerformance  Appraisal  activities  had  a  significant  number  of 
problems.  Table  10  shows  the  results  of  Task  13,  opening  the  Annual  Performance  Appraisal  1 
document.  There  were  17  incidences  of  navigational  errors  opening  the  document,  and  one  user 
asked  for  help.  It  is  not  surprising  that  users  continued  to  have  problems  navigating  through 
PeopleSoft  to  identify  the  appropriate  performance  appraisal  document.  The  most  frequent 
problems  observed  among  the  HPFD  tasks  were  finding  the  HPFD  document  and  finding  the 
performance  notes  document.  It  appears  that  opening  the  performance  appraisal  was  not  much 
easier.  Navigation  within  PeopleSoft  is  clearly  a  challenge  to  users.  Users  of  ePerformance 
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Appraisal  system  would  likely  benefit  from  a  QRG  similar  to  the  one  developed  for  HPFD.  The 


remaining  ePerformance  Appraisal  system  task  details  are  presented  in  Appendix  B. 

Table  10.  Task  13:  Open  the  Annual  Performance  Appraisal  1  document 


Supervisors  | 

Total  Number  of  Incidents 

Average  Number  of  Incidents 

Problem  Category 

Across  Sessions 

Per  Session 

Navigational  error 

17 

5.66 

User  asks  for  help 

1 

1 

6.3  Focus  Group  Results 

Focus  group  results  were  divided  into  the  two  main  groups  of  study  participants — 
supervisors  and  non-supervisors.  In  this  report,  comments  are  summarized  in  text,  and  the  main 
qualitative  themes  and  frequency  of  a  comment  are  displayed  in  tables.  Complete  participant 
comments  (color-coded  to  illustrate  comments  made  at  each  data  collection  site)  can  be  found  in 
the  appendices  (supervisor  quotes  in  Appendix  C;  non- supervisory  quotes  in  Appendix  D ). 

6.3.1  Supervisory  Personnel 

Supervisory  personnel  at  each  of  the  three  sites  participated  in  focus  group  interviews  and 
were  asked  about  the  phases  involved  in  using  the  HPFD  and  ePerformance  systems.  Main 
themes  and  the  frequency  comments  in  each  theme  category  are  shown  in  Table  11. 

NPC  provided  a  CBT  module  to  assist  participants  in  understanding  key  terms  and  HPFD 
system  features.  The  CBT  consisted  of  a  skills  pre-test,  a  training  module  on  how  to  use  the 
HPFD  CBT  tutorial,  and  the  HPFD  CBT  tutorial.  The  HPFD  CBT  tutorial  consists  of  nine 
sections:  HPFD  Overview,  Security  and  Account  Creation,  Human  Resources  Administrative 
Functions,  Manager  Functions,  Employee  Functions,  Rating  Categories  and  Target  Behaviors, 
Performance  Notes,  Lesson  Summary,  and  a  Lesson  Quiz.  Usability  testing  participants  were 
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Table  11.  Supervisory  Personnel — Summary  of  Major  Themes  and  Frequency  of  Comments  for  Each  Phase  of  HPFD  and 
ePerformance  Process 


Focus  Group 
Interview  Topic 

What  Worked  Well  -  Major  Themes 

Frequency 

Opportunities  for  Growth  -  Major  Themes 

Frequency 

Simplicity  of  Learning  Process/Didn’t  Relay 

11 

Training  and  Reference 

General  Satisfaction 

5 

Document  Navigation  Information 

Materials  -  CBT 

Clarity  of  Information 

3 

Relevance 

1 

Training  and  Reference 

Availability  &  Accessibility 

2 

Materials  -  Quick 
Reference  Guide 

Usefulness  of  QRG 

9 

Presentation  of  Information  -  Relevance 

2 

Logging  Onto  the  System 

Ease  of  Login  (When  System  was  Functioning) 

6 

Trouble  Navigating  to  the  Document 

13 

Unclear  Password  Requirements 

6 

Ease  of  Selecting  the  Document  Facilitated  by  QRG 

8 

Login  Failures  Due  to  NSIPS  Problems 

6 

Selecting  and  Opening 

Trouble  Identifying  the  Document 

5 

Your  Document 

Document  was  Easy  to  Find 

5 

Page  Loading 

1 

Terminology  “Too  Civilian” 

1 

General  Positive  Comments 

6 

Paper  Document  Back-up 

10 

Performance  Dimensions  -  Well  Phrased 

5 

NSIPS  Problems 

7 

Completing  Your  HPFD 
Form 

HPFD  Process 

2 

Performance  Dimensions  -  Repetitive  &  “Too 

Civilian” 

6 

Web-based  Form 

5 

HPFD  Process  -  Need  for  “Face  to  Face”  Interaction 

6 

Process  Ownership 

1 

Navigation  and  Process  Clarification  Needed 

6 

Text  Box  Limitations  Necessary 

3 

Easy  to  Use 

Cultural  Issues  -  Promotion  Recommendation  & 

2 

Forced  Distribution 

7 

Completing  Your 
ePerformance  Form 

Form  Design  -  Text  Boxes  for  Each  Performance 
Dimension  &  Overall  Summary  Block 

6 

ePerformance  Process  Concerns 

7 

Cultural  Issues  -  Text  Boxes 

6 

Clarity  of  Performance  Dimensions 

3 

Performance  Dimensions  -  Repetitive  &  “Too 

Civilian” 

b 
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Table  11.  Supervisory  Personnel — Summary  of  Major  Themes  and  Frequency  of  Comments  for  Each  Phase  of  HPFD  and 
ePerformance  Process  (continued) 


Focus  Group 
Interview  Topic 

What  Worked  Well  -  Major  Themes 

Frequency 

Opportunities  for  Growth  -  Major  Themes 

Frequency 

Text  Box  Limitations  Necessary 

6 

Save  and  Complete/Forward  Documentation 

5 

Closing  your  HPFD  or 
ePerformance  Session 

General  Positive  Comments 

2 

Form  Routing 

Paper  Document  Back-up 

4 

3 

Forced  Distribution/Promotion  Summary 
Recommendation  Concerns 

3 

Spell  Check/Language  Check 

3 

Workflow  Concerns:  First-level  Rater  &  Form 
Routing 

10 

Concerns  About  Procedural  Fairness 

8 

Other  Concerns 

Performance  Notes 

2 

Implementation  Concerns 

6 

Forced  Distribution  Questions 

6 

Connectivity  Concerns 

6 

Resistance  to  Change 

1 
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asked  to  complete  only  the  HPFD  CBT  tutorial  in  order  to  familiarize  themselves  with  the 
functions  of  the  HPFD  system.  When  asked  about  the  CBT,  most  participants  expressed 
dissatisfaction  with  the  CBT.  The  most  frequently  cited  criticism  of  the  CBT  was  a  general  lack 
of  simplicity  in  relaying  the  information  about  the  HPFD  system.  Some  typical  comments 
include  “Learning  objectives  were  not  obvious;  it  provided  a  familiarity  rather  than  describing 
how  to  use  the  system”  and  “CBT  did  not  help  me  find  the  documents;  it  only  helped  with  filling 
out  the  documents.” 

As  a  result  of  comments  in  the  first  iteration  of  the  study  (i.e.,  “Sailors  work  from 
checklists — they’re  used  to  that;  the  training  should  be  set  up  like  a  checklist  to  work  people 
through  the  training”),  the  research  team  developed  a  Quick  Reference  Guide  (QRG)  that 
included  simple  screenshots  and  caption  boxes  that  illustrated  where  documents  and  key 
functions  could  be  found.  Supervisors  generally  found  the  QRG  useful,  citing  that  “this  is  [the 
QRG]  critical”  and  that  the  “QRG  is  easy  to  understand.”  Participants  stated  that  the  QRG  could 
be  improved  if  functions  and  critical  features  of  the  program  were  highlighted  and  presented  in  a 
sequential  fashion  that  leads  participants  through  HPFD  and  ePerformance  tasks. 

When  asked  about  logging  onto  the  system,  several  supervisors  reported  that  it  was  an 
easy  process  when  the  NSIPS  system  was  functioning.  The  NSIPS  system  had  recurring 
interruptions  in  service,  which  led  to  a  great  deal  of  frustration  among  participants.  Beyond  the 
NSIPS  system  challenges,  participants  cited  two  aspects  of  the  password  change  process  that 
should  be  improved:  making  the  password  non-case  sensitive  and  outlining  the  password 
requirements  when  first  asked  to  reset  passwords  (i.e.,  one  upper-case  character,  one-lower  case 
character,  minimum  password  length,  special  characters,  and  numeric  characters).  As  one 
participant  commented,  “The  rules  for  password  generation  should  be  published.”  Frequently, 
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users  learned  that  their  passwords  did  not  satisfy  the  password  requirements  only  after  several 
trial-and-error  attempts. 

Supervisors  were  then  asked  about  their  experience  selecting  and  opening  their  HPFD 
and  ePerformance  documents.  In  the  first  iteration  (without  the  benefit  of  the  QRG),  several 
users  reported  difficulty  knowing  where  to  find  the  HPFD  or  ePerformance  document,  but  once 
found,  the  hyperlink  to  the  document  was  easy  to  select.  Eight  of  13  comments  describing 
problems  finding  the  document  (navigating  the  system)  were  from  supervisors  in  Iteration  1: 

“. . .  [Wjhere  do  I  start  once  I  get  logged  in?  Navigation  is  not  intuitive  and  system  was  not 
explained  well  in  the  CBT.”  Those  supervisors  in  Iterations  2  and  3  who  used  the  QRG  indicated 
that  it  aided  them  in  locating  and  completing  the  documents:  “I  used  the  [QRG]  and  whipped 
right  through  the  process.” 

When  completing  the  HPFD  form,  participants  provided  general  positive  comments: 

“The  form  was  easy  to  complete  and  fill-in  the  block  with  comments;  it  was  simple  and 
efficient.”  The  terminology  used  with  the  performance  dimensions  were  “well  organized  and 
worded  logically”  and  reflected  “phrases  I’d  use  in  the  written  text  of  a  FITREP  [senior  enlisted 
and  officer  performance  appraisal]  or  EVAL  [junior  to  mid-grade  performance  appraisal].” 

When  asked  for  features  of  the  HPFD  system  that  could  be  improved,  supervisors  cited  the  lack 
of  an  electronic  or  paper  back-up  option,  NSIPS  system  performance,  the  performance 
dimensions,  and  concerns  with  the  HPFD  process.  Supervisors  felt  that  a  paper  back-up  copy 
would  be  helpful  for  as  insurance  against  document  loss  in  the  event  of  a  system  failure,  and  so 
Sailors  could  work  when  Internet  access  is  unavailable  either  due  to  the  scarcity  of  computers 
aboard  ship  or  when  Internet  access  is  limited  while  ships  and  submarines  are  at  sea. 
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While  the  terminology  used  for  the  performance  dimensions  was  mentioned  as  a  positive 
feature  of  the  HPFD  system,  several  supervisors  felt  that  the  terminology  was  “too  civilian”  and 
that  several  of  the  performance  dimensions  seemed  redundant.  Additionally,  concern  about  how 
the  HPFD  process  would  be  implemented  in  a  Web-based  environment  led  to  comments  stating 
“I  want  to  do  a  face-to-face  counseling  with  Sailors  first,  complete  the  counseling  document  and 
send  it  to  the  Sailor  for  review,  and  they  can  call  if  they  have  any  problems.”  Participants  at  each 
testing  site  echoed  this  sentiment. 

When  describing  their  experience  in  completing  the  ePerformance  document,  supervisors 
reported  that  the  Web-based  form  with  text  boxes  for  each  performance  dimension  was  easy  to 
use  and  that  “. .  .a  separate  comment  box  for  each  behavior  is  ideal.”  Also,  participants  said  that 
the  performance  dimensions  were  clear  and  made  it  easier  to  rate  workplace  performance:  “Each 
behavior  description  is  a  lot  clearer,  easier  to  rate  people  than  NAVFIT98,  and  descriptions  of 
anchors  were  good.” 

Aspects  of  the  ePerformance  system  that  drew  criticism  were  the  absence  of  the  reporting 
senior’s  promotion  recommendation  (i.e.,  “forced  distribution”),  process  concerns,  concerns 
regarding  the  amount  of  and  use  of  space  in  each  text  box,  and  the  perception  that  several  of  the 
performance  dimensions  were  redundant.  Many  of  the  comments  pertaining  to  the  promotion 
summary  block  more  truly  reflect  questions  about  how  the  Navy  will  identify  and  compare 
individuals  for  promotion:  “How  will  ‘top  performers’  be  given  the  opportunity  to  take  an 
advancement  test  early?”  Process  concerns  also  reflected  questions  about  how  the  current  Navy 
performance  appraisal  process  would  be  adapted  to  align  with  other  Navy  manpower  and 
personnel  processes  (i.e.,  selection  boards,  detaching/special  FITREPs/EVALs,  and  procedures 
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for  Sailors  to  disagree  with  the  performance  appraisal  and  submit  a  statement  contesting  the 
appraisal). 

When  closing  the  HPFD  or  ePerformance  document,  several  supervisors  repeated 
concerns  mentioned  in  previous  phases  of  system  use  (i.e.,  electronic  or  paper  back-up  copies, 
text  box  limitations).  Other  concerns  not  previously  mentioned  included  confusion  between  the 
“Save”  and  “Complete”  buttons  and  questions  pertaining  to  document  routing  and  processing 
(“What  is  the  final  disposition  of  your  ePerformance  document?”).  Users  reported  uncertainty  as 
to  what  differentiated  the  “Save”  and  “Complete”  functions.  Several  participants  recommended  a 
confirmation  screen  that  asks  system  users  to  confirm  their  intent  (i.e.,  “Pressing  ‘Yes’  will  save 
the  document  and  forward  it  to  your  supervisor  for  review  and  processing.”). 

Finally,  participants  were  asked  if  there  were  any  additional  comments  or  concerns  that 
they  would  like  to  convey.  For  supervisory  Sailors,  a  number  of  implementation  and  process 
concerns  emerged — at  what  level  in  the  organization  would  the  first-level  rater  be  given  the 
responsibility  for  the  performance  appraisal,  the  notion  of  the  first-level  rater’s  comments 
remaining  unchanged  as  the  performance  appraisal  is  routed  up  the  chain  of  command,  concerns 
of  procedural  fairness  and  grade  inflation,  and  the  impact  of  removing  the  reporting  senior’s 
promotion  recommendation. 

Supervisors  were  then  asked  to  sum  up  their  experience  by  grading  the  system  on  a  scale 
of  0  (low)  to  4  (high).  Scores  for  supervisors  were  below  the  scale  mid-point  with  the  NAS 
Brunswick  supervisors  rating  the  HPFD  and  ePerformance  systems  lowest  ( X  =  1.70)  and 
supervisors  from  Naval  Base  Kitsap  -  Bangor  rating  the  system  highest  ( X  =  2.25;  See 
Table  12).  Factors  contributing  to  subjective  ratings  include  problems  with  the  NSIPS  system 
and  interruptions  in  connectivity.  Also,  system  users  appeared  to  have  a  number  of  process- 
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related  questions  (i.e.,  workflow,  who  the  first-level  rater  would  be,  and  form  routing)  and 

implementation  questions  that  could  have  negatively  affected  subjective  ratings  (See  Table  13). 

Table  12.  Supervisory  Personnel — Overall  Subjective  Grade  of  HPFD  and  ePerformance 
Processes  and  Systems. 


Grading  the  System 

Average  Grade* 

NAS  Brunswick, 

Brunswick  Maine 

1.70 

USS  KITTY  HAWK  (CV63) 

Yokosuka,  Japan 

1.75 

Kitsap  Naval  Base  -  Bangor 

Bangor,  WA 

2.25 

Note:  Average  Grade  rated  on  a  0  (low)  -  4  (high)  scale. 


Table  13.  Supervisory  Personnel — Overall  Subjective  Grade  of  HPFD  and  ePerformance 
Processes  and  Systems. 


Reasons  for  Grade 

Frequency 

NSIPS/Connectivity  Concerns 

12 

Advantages  over  NAVFIT98 

9 

Clarity  of  Instructions  and  Procedures 

5 

System  Design  and  Interface  Improvements  Necessary 

5 

Performance  Appraisal  Process  Concerns 

3 

Internet  Access 

3 

Administrative  Burden 

2 

6.3.2  Non-supervisory  Personnel 

Similar  to  the  procedure  for  supervisory  personnel,  non-supervisory  personnel  at  each  of 
the  three  sites  participated  in  focus  group  interviews  and  were  asked  about  the  phases  involved  in 
using  the  HPFD  systems.  Sailors  in  non-supervisory  positions  were  not  asked  to  complete  an 
ePerformance  document,  since  supervisors  are  typically  assigned  with  conducting  annual 
performance  appraisals.  Participant  comments  were  sorted  into  main  themes  for  each  phase  in 
the  document  use  and  completion  process.  Main  themes  and  the  frequency  comments  in  each 
theme  category  appear  in  Table  14. 
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Table  14.  Non-supervisory  Personnel — Summary  of  Major  Themes  and  Frequency  of  Comments  for  Each  Phase  of  HPFD 
and  ePerformance  Process. 


Focus  Group  Interview 

Topic 

What  Worked  Well  -  Major  Themes 

Frequency 

Opportunities  for  Growth  -  Major  Themes 

Frequency 

Training  and  Reference 
Materials  -  CBT 

General  Satisfaction 

1 

Simplicity  of  Learning  Process/Didn’t  Relay 

Document  Navigation  Information 

11 

Clarity  of  Information 

2 

Training  and  Reference 
Materials  -  Quick 

Usefulness  of  QRG 

7 

Presentation  of  Information  -  Include  Buttons/Icons 
in  QRG 

1 

Reference  Guide 

Logging  Onto  the  System 

Ease  of  Login  (When  System  was  Functioning) 

5 

Unclear  Password  Requirements 

10 

Selecting  and  Opening 
Your  Document 

Ease  of  Selecting  the  Document  Facilitated  by  QRG 

10 

Trouble  Identifying  the  Document 

1 

Trouble  Navigating  Through  the  Document 

5 

Performance  Notes 

4 

Paper  Document  Back-up 

4 

Process  Ownership 

4 

Internet/NSIPS  Connectivity 

3 

Completing  Your  HPFD 

Form 

Performance  Dimensions  -  Well  Phrased 

2 

Implementation  Concerns 

2 

Web-based  Form 

2 

HPFD  Process  -  Need  for  “Face  to  Face”  Interaction 

1 

General  Positive  Comments 

1 

Closing  your  HPFD  or 
ePerformance  Session 

Save  and  Complete/Forward  Documentation 

5 

Performance  Notes 

1 

HPFD  Process 

5 

Paper  Document  Back-up 

2 

HPFD  Process:  Face-to-Face 

5 

Other  Concerns 

Performance  Notes 

2 

NSIPS/Internet  Connectivity  Concerns 

4 

System  Training  Concerns 

3 
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Much  like  their  supervisory  counterparts,  many  of  these  non- supervisory  Sailors  were 
asked  to  view  the  CBT  and/or  use  the  QRG.  Non- supervisory  Sailors  reported  that  the  CBT 
failed  to  simply  and  concisely  review  the  background  on  HPFD  and  clearly  present  the  learning 
objectives  for  the  HPFD  system:  “Learning  objectives  not  clear;  need  to  simplify  CBT.”  System 
users  also  reported  that  terminology  and  acronyms  could  have  been  presented  more  clearly.  Non- 
supervisory  Sailors  were  provided  with  the  QRG  to  aid  in  using  the  HPFD  and  ePerformance 
systems.  For  the  QRG,  most  comments  relayed  general  satisfaction  with  the  guide:  “After 
receiving  the  QRG,  I  could  have  written  an  HPFD  document  without  a  problem.” 

When  reporting  on  their  experience  logging  onto  the  HPFD  system,  several  participants 
reported  that  the  login  process  worked  smoothly.  The  vast  majority  of  non- supervisory  Sailors 
commented  that  the  aspect  of  the  login  process  that  needed  the  most  improvement  was  how  a 
system  user  changes  their  passwords  and  how  the  password  requirements  are  displayed.  Similar 
to  supervisory  Sailors,  non-supervisory  Sailors  would  like  to  view  that  information  at  the  same 
time  that  a  replacement  password  is  entered. 

Non-supervisory  Sailors  reported  that  the  QRG  was  useful  in  selecting  and  opening  their 
HPFD  document.  Several  participants  reported  dismissing  the  PeopleSoft  terminology  that 
appeared  on  the  program’s  navigation  section,  relying  entirely  on  the  information  and  screen- 
shots  illustrated  in  the  QRG.  Half  of  those  participants  who  had  problems  selecting  and  opening 
their  document  were  non-supervisory  Sailors  from  NAS  Brunswick — the  only  site  that  did  not 
have  access  to  the  QRG. 

When  using  the  HPFD  system,  non-supervisory  Sailors  relayed  a  sense  of  process 
ownership  because  the  HPFD  system  allowed  the  user  to  examine  their  5-Vector  Model  and 
identify  specific  performance  dimensions  for  continued  development  in  order  to  enhance  their 
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careers.  In  conjunction  with  the  notion  of  process  ownership,  users  remarked  that  the 
performance  notes  feature  appeals  to  them  because  they  can  track  significant  performance 
accomplishments  throughout  the  year.  The  most  frequently  occurring  recommendation  to 
improve  the  HPFD  was  to  include  the  capability  for  a  printed  back-up  copy.  Other 
recommendations  for  improving  the  HPFD  system  included  placing  an  NSIPS  server  aboard 
ships  and  submarines  so  HPFD  could  be  run  off  of  the  ship’s  or  submarine’s  intranet.  For  surface 
ships,  this  is  important  since  bandwidth  for  Internet  traffic  is  limited,  and  aboard  submarines, 
Internet  connectivity  is  scarce. 

When  asked  about  concluding  their  HPFD  session,  non- supervisory  Sailors  noted 
challenges  determining  which  button  closed  out  the  document:  “. . .  ‘complete’  button  is  too  easy 
to  confuse  with  ‘save’  button;  ‘complete’  button  needs  a  confirmation  pop-up  [to  give  additional 
directions].”  Other  comments  cited  echo  concerns  previously  mentioned  for  a  paper  back-up 
copy  of  the  HPFD  form  and  hosting  the  NSIPS  server  on  the  ship’s  intranet  to  ensure  system 
access. 

Finally,  participants  were  asked  if  there  were  any  additional  comments  or  concerns  that 
they  would  like  to  convey.  For  non-supervisory  Sailors,  the  notion  of  how  the  HPFD  session 
would  be  conducted  was  of  great  concern.  While  there  were  differences  in  who  would  initiate  the 
HPFD  session  (e.g.,  Sailor  complete  a  self-assessment,  supervisor  complete  a  draft  HPFD 
assessment),  all  agreed  that  a  face-to-face  performance  feedback  session  ought  to  be  included  in 
the  HPFD  process  to  maintain  personal  contact  with  their  supervisors.  Non-supervisors  were 
then  asked  to  sum  up  their  experience  by  grading  the  system  on  a  scale  of  0  (low)  to  4  (high). 
Scores  for  non- supervisors  were  at  or  above  the  scale  mid-point  with  the  NAS  Brunswick  non¬ 
supervisors  rating  the  HPFD  and  ePerformance  systems  lowest  ( X  =  2.00),  with  ratings  from 
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Naval  Base  Kitsap  -  Bangor  rating  the  system  highest  ( X  =  2.75;  see  Table  15).  Factors 
contributing  to  subjective  ratings  include  a  lack  of  instructions  for  system  use  and  problems  with 


the  NSIPS  system  and  interruptions  in  connectivity  (See  Table  16). 

Table  15.  Non-supervisory  Personnel — Overall  Subjective  Grade  of  HPFD  and 
ePerformance  Processes  and  Systems 


Grading  the  System 

Average  Grade* 

NAS  Brunswick, 

Brunswick  Maine 

2.00 

USS  KITTY  HAWK  (CV63) 

Yokosuka,  Japan 

2.17 

Kitsap  Naval  Base  -  Bangor 

Bangor,  WA 

2.75 

Note:  Average  Grade  rated  on  a  0  (low)  -  4  (high)  scale. 


Table  16.  Non-supervisory  Personnel — Overall  Subjective  Grade  of  HPFD  and 
ePerformance  Processes  and  Systems. 


Reasons  for  Grade 

Frequency 

NSIPS/Connectivity  Concerns 

3 

Clarity  of  Instructions  and  Procedures 

4 

Performance  Appraisal  Process  Concerns 

5 

General  Satisfaction 

2 
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7  Summary  and  Conclusions 


7.1  Key  Findings 

Taken  as  a  whole,  results  indicate  that  the  HPFD  and  ePerformance  systems  themselves 
worked  well.  User  ratings  of  the  system  and  qualitative  data  from  focus  group  interviews 
indicate  that  system  users  believe  the  systems  work  well,  with  a  majority  of  users  reporting  that 
the  system  “was  neither  easy  nor  difficult  to  use.”  Unfortunately,  the  same  proportion  of  users 
felt  the  system  was  “somewhat  difficult  to  use”  (23  percent)  or  “somewhat  easy  to  use”  (23 
percent).  Qualitative  data  from  focus  group  interviews  indicate  that  the  frequency  and  severity  of 
NSIPS  connectivity  problems  had  a  significant  negative  effect  on  user  perceptions  of  the  HPFD 
and  ePerformance  systems.  In  spite  of  system  connectivity  problems,  user  ratings  and  feedback 
show  promise  for  these  systems  still  in  the  early  phase  of  development. 

While  the  goal  of  this  usability  study  was  to  examine  system  usability,  identify  areas  for 
concern,  and  form  recommendations  for  revision,  we  uncovered  additional  usability  concerns 
that  play  a  key  role  in  the  successful  implementation  of  the  HPFD  and  ePerformance  system. 
System  connectivity  (i.e.,  NSIPS  problems  and  internet  connectivity)  and  cultural/process 
concerns  emerged  as  other  features  that  warrant  further  attention.  Unfortunately,  we  were  unable 
to  test  other  key  system  features  in  this  phase  of  the  study,  namely  document  workflow. 

7.1.1  NSIPS  and  Internet  Connectivity  Concerns 

NSIPS  connectivity  and  server  downtime  were  significant  problems  for  users  in  actually 
using  the  HPFD  and  ePerformance  systems  and  completing  assigned  usability  tasks  (i.e.,  slow 
page-loading  time,  system  timing  out,  passwords  working  inconsistently).  Additionally, 
participant  subjective  ratings  of  their  satisfaction  with  the  systems  indicate  that  these  problems 
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negatively  affected  their  satisfaction  with  the  HPFD  and  ePerformance  systems. 
Recommendations  to  alleviate  these  problems  may  include: 

•  Placing  NSIPS  servers  at  the  local  command,  ship,  or  submarine  so  that  (1) 
information  transactions  can  be  completed  on  a  local  intranet  before  data  are 
submitted  via  the  Internet  to  the  main  NSIPS  server,  and  2)  Sailors  aboard  any 
platform  ship  or  submarine  do  not  feel  alienated  and  at  a  competitive  disadvantage  to 
shore -based  Sailors  in  the  performance  appraisal  process;  and 

•  Addressing  the  problem  of  timing  out  by  adding  additional  “save”  buttons  within  the 
HPFD  and  ePerformance  systems,  creating  an  “autosave”  function,  adding  a  “timeout 
indicator”  for  the  user  to  gauge  time  remaining  until  a  system  timeout,  or  re-setting 
the  system  timeout  parameter  to  a  reasonably  large  enough  time  value  for  users  to 
complete  and  save  their  work. 

7.1.2  Usability  Errors 

There  are  a  number  of  features  within  the  HPFD  and  ePerformance  systems  that  were 
identified  as  sources  of  error  and  frustration  for  system  users.  Making  some  simple  corrections  to 
the  system  could  markedly  improve  the  usability  of  the  systems.  Recommendations  for  action 
include: 

•  Adding  a  “Return  to  Document”  button  for  returning  to  the  main  screen,  minimizing 
the  likelihood  that  users  will  use  the  browser’s  “back”  button  to  navigate  the  system; 

•  Simplifying  and  more  clearly  outlining  the  password  change  requirements  when  the 
participant  is  first  asked  to  change  their  password; 
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•  Clarifying  buttons/icons  by  including  descriptive  verbal  labels  for  “Spell  Check,” 
“Save,”  “Expand/Collapse  All,”  “Target  Behavior  Descriptions,”  and  “Complete” 
buttons; 

•  Adding  a  confirmation  screen  for  the  “Save”  and  “Complete”  buttons  that  asks  the 
user  to  confirm  that  “Pressing  ‘YES’  will  save  and  forward  this  document  to  your 
supervisor  for  processing”; 

•  Moving  the  “Search  Documents”  box  below  the  “In  Progress  Documents”  display  so 
users  avoid  using  the  search  feature  in  favor  of  using  the  navigation  menu  on  the  left¬ 
side  of  the  system  screen;  and 

•  Creating  a  “Quick  Reference  Guide”  with  screen-shots  and  descriptive  text  boxes  (to 
replace  the  CBT)  that  provide  a  step-by-step  procedure  for  creating,  completing,  and 
approving  HPFD  and  ePerformance  documents. 

7.1.3  Cultural  and  Process  Concerns 

From  the  wealth  of  qualitative  data  gathered  from  usability  study  participants,  a  number 
of  recurring  themes  highlighted  cultural  and  process  concerns  users  had  with  the  systems.  While 
the  information  participants  had  of  the  proposed  revision  to  the  Navy  performance  counseling 
and  appraisal  system  varied  among  test  sites,  the  following  similar  themes  emerged: 

•  Include  a  hard-copy  back-up  of  the  HPFD  and  ePerformance  document  (e.g.,  an 
Adobe  Acrobat  Portable  Document  Format  or  PDF  form)  to  (1)  serve  as  a  personal 
back-up  in  the  event  of  a  system  failure  and  information/document  loss,  and  (2) 
enable  Sailors  with  limited  online  system  access  to  develop  HPFD  or  ePerformance 
input  on  a  hard-copy  form  and  then  enter  the  data  online  when  they  have  access  to  the 
system; 


42 


•  Conduct  follow-up  focus  group  interviews  to  examine  cultural  issues  related  to  the 
proposed  new  HPFD  and  ePerformance  systems;  determine  at  what  level  within  the 
organization  HPFD  and  ePerformance  appraisals  can  be  delegated;  examine  more 
closely  the  implications  of  the  number  and  amount  of  text  allowed  for  separate  text 
boxes  for  each  performance  dimension,  and  evaluate  the  proposed  period  of 
performance  counseling  and  appraisal  based  on  report  onboard  date; 

•  Ensure  that  the  HPFD  and  ePerformance  systems  are  integrated  into  a  Navy  culture 
that  encourages  strong,  personal  and  professional  relationships  between  work-center 
supervisors  and  their  subordinates,  including  as  much  personal,  face-to-face 
interaction  in  the  process  as  possible; 

•  Develop  a  communication  plan  that  concisely  outlines  the  rationale  and  benefit  of 
proposed  changes  to  the  performance  counseling/appraisal  systems  and  that  describes 
how  data  from  the  new  HPFD  and  ePerformance  systems  will  be  integrated  into  Navy 
manpower  and  personnel  processes  (i.e.,  selection  and  advancement  boards, 
personnel  assignment,  and  distribution  systems,  etc);  and 

•  Follow  up  with  key  process  stakeholders  on  many  of  these  cultural  and  process 
changes  with  a  goal  of  identifying  challenges  and  solutions  for  system 
implementation. 

7.2  Limitations  of  research 

There  are  several  limitations  typical  of  usability  studies  including  small  sample  sizes  and 
generalizability/representativeness.  First,  a  total  of  57  Sailors  participated  in  the  usability  testing 
across  the  three  iterations.  While  this  may  seem  like  too  few  subjects  from  which  to  generalize, 
most  usability  studies  test  a  much  smaller  group  of  subjects.  Dolan  and  Dumas  (1999) 
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recommend  testing  between  five  and  10  participants  per  iteration.  The  current  study  doubled  the 
number  of  participants  per  iteration  in  order  to  get  an  adequate  number  of  supervisory  and  non- 
supervisory  Sailors  testing  the  HPFD  and  ePerformance  systems. 

Second,  because  of  the  qualitative  nature  of  this  study,  results  are  not  representative  of  all 
Navy  personnel.  The  study  design  does  capture  a  meaningful  representation  of  system  users  both 
in  terms  of  demographic  characteristics  (i.e.,  race,  gender,  paygrade,  geographic  location)  and 
workplace  characteristics  (i.e.,  supervisory  and  non-supervisory  personnel,  Navy  warfare 
communities — aviation,  surface,  and  submarine,  administrative  and  non-administrative  jobs). 

Finally,  most  usability  testing  study  designs  call  for  an  iterative  approach  in  which 
revisions  are  made  between  iterations.  This  study  was  limited  in  the  number  and  types  of 
changes  between  iterations.  The  initial  study  design  called  for  system  modifications  between 
iterations.  During  the  first  iteration  of  this  study,  the  research  team  learned  that  this  design  aspect 
could  not  be  accommodated. 

7.3  Recommendations  for  future  research 

Results  from  this  study  raise  several  questions  that  could  be  addressed  through  future 
research.  First,  while  the  recommendations  from  this  study  are  likely  to  be  incorporated  in 
subsequent  versions  of  the  Navy  HPFD  and  ePerformance  systems,  a  small  follow-up  usability 
study  may  need  to  be  conducted  to  confirm  that  changes  made  to  the  system  indeed  made  it  a 
more  usable  system — fewer  NSIPS  problems,  decreased  time  spent  on  completing  tasks,  and 
increased  satisfaction  with  the  system. 

Second,  the  next  phase  of  this  study  would  be  to  conduct  a  full  pilot  study  where  an  entire 
command  would  complete  the  performance  counseling  and  appraisal  process  using  the  HPFD 
and  ePerformance  systems.  This  study  did  not  test  the  task  from  its  true  beginning — an  e-mail 
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notification  that  a  performance  document  needed  to  be  created,  creating  the  document,  soliciting 
performance  input,  completing  the  input,  and  routing  it  through  the  unit/command  for  approval. 

A  full  pilot  study  would  test  the  impact  of  conducting  performance  appraisals  throughout  the 
year  (based  on  one’s  date  of  reporting  to  the  command)  versus  the  current  system  of  conducting 
performance  appraisals  in  certain  pre-specified  periods  (e.g.,  one  paygrade  receives  performance 
appraisals  in  September,  another  paygrade  in  October,  and  so  on). 

Finally,  results  from  participant  focus  group  interviews  raised  questions  in  terms  of  how 
the  HPFD  and  ePerformance  systems  would  be  implemented  (i.e.,  fear  of  a  loss  of  personal 
contact  between  supervisor  and  subordinate  using  Web-based  performance  management  systems, 
senior  leadership’s  inability  to  have  a  strong  influence  on  written  text  that  substantiates  a 
performance  appraisal  rating,  concerns  about  the  amount  of  space  available  for  written  text,  and 
fears  that  this  will  burden  those  completing  performance  appraisals).  These  and  other  cultural 
concerns  should  be  studied  with  focus  group  interviews  with  recommendations  being 
incorporated  into  the  final  performance  appraisal  process. 


45 


8  Closing 


This  study  provides  information  that  will  significantly  improve  the  Navy’s  HPFD  and 
ePerformance  system  and  its  implementation.  As  mentioned  earlier,  active  participation  in  the 
performance  appraisal  process  is  a  significant  component  of  job  satisfaction,  organizational 
commitment,  and  a  likely  factor  in  retention  plans.  Involving  Navy  personnel  at  each  step  in 
developing  the  HPFD  and  ePerformance  system  is  crucial  to  stakeholder  acceptance  and  buy-in. 
The  development  of  the  HPFD  and  ePerformance  system  involved  Navy  personnel  from  junior 
paygrades  to  senior  leadership  in  the  identification  of  supervisory  and  non-supervisory 
performance  dimensions  (Hedge,  Borman,  Bruskiewicz,  &  Bourne,  2002);  identifying  examples 
of  workplace  behaviors  that  reflect  outstanding,  average,  and  substandard  workplace 
performance  (Borman,  Hedge,  Bruskiewicz,  &  Bourne  2003);  conducting  focus  groups  among 
senior  leaders  to  weight  the  various  performance  dimensions  (Hedge,  Bruskiewicz,  Borman,  & 
Bourne,  2004);  and  the  current  study  that  tested  the  usability  of  the  HPFD  and  ePerformance 
systems.  This  “fleet-driven”  approach  will  provide  a  more  credible,  usable,  and  likely  more 
easily  accepted  process  that  is  a  vast  departure  from  the  current  performance  appraisal  process. 

While  more  work  is  required  to  further  refine  and  test  the  HPFD  and  ePerformance 
systems,  these  systems  represent  a  significant  step  forward  in  making  a  more  objective  and  fair 
performance  appraisal  system.  With  all  of  the  sacrifices  military  personnel  make  in  their  national 
service,  they  deserve  a  performance  appraisal  system  that  is  fair,  equitable,  and  easy  to  use.  This 
reflects  the  CNO’s  emphasis  on  covenant  leadership,  “I  see  a  Navy  in  which  all  our  leaders  are 
personally  committed,  first  and  foremost,  to  mission  accomplishment...  and  second,  to  the 
growth  and  development  of  the  people  who  are  entrusted  to  us.  This  is  part  of  the  covenant — the 
promise — of  leadership.  And  this  is  very  important  to  me.  I  want  every  one  of  our  jobs  to  be  full 
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of  meaning  and  purpose.  I  want  every  one  of  our  people  in  the  Navy  to  have  a  rich  and 
meaningful  experience”  (Clark,  2001). 
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Appendix  A 
Questionnaire  Data 


Pretest  Survey 

Fifty- seven  participants  completed  the  pretest  survey. 


Q1 

How  old  are  you? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

18-24 

12 

21 .05% 

2 

5.71% 

10 

45.45% 

2 

25-34 

25 

43.86% 

16 

45.71% 

11 

50.00% 

3 

35-44 

18 

31 .58% 

16 

45.71% 

2 

9.09% 

4 

45-54 

2 

3.51% 

2 

5.71% 

0 

0.00% 

5 

55+ 

0 

0.00% 

0 

0.00% 

0 

0.00% 

Q2 

What  is  your  gender? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Male 

51 

89.47% 

31 

88.57% 

20 

90.91% 

2 

Female 

6 

10.53% 

4 

1 1 .43% 

2 

9.09% 

Q3 

Are  you  of  Spanish, 
Hispanic,  or  Latino 
origin? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Yes 

4 

7.02% 

1 

2.86% 

3 

13.64% 

2 

No 

53 

92.98% 

34 

97.14% 

19 

86.36% 

Q4 

What  is  your  racial 
background?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

American  Indian  or  Alaska 
Native 

3 

5.26% 

1 

2.86% 

2 

9.09% 

2 

Asian 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

3 

Black  or  African-American 

6 

10.53% 

2 

5.71% 

4 

18.18% 

4 

Native  Hawaiian  or  other 
Pacific  Islander 

0 

0.00% 

0 

0.00% 

0 

0.00% 

5 

White 

43 

75.44% 

28 

80.00% 

15 

68.18% 

6 

Other 

5 

8.77% 

1 

2.86% 

4 

18.18% 

*The  overall  count  is  higher  than  57  because  several  respondents  reported  multiple  races. 


A-l 


Q5 

What  is  the  highest 
level  of  education  you 
have  completed? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Less  than  high  school 
completed/no  diploma 

0 

0.00% 

0 

0.00% 

0 

0.00% 

2 

Alternate  degree,  GED, 
homestudy,  or  adult- 
school  certification 

3 

5.26% 

1 

2.86% 

2 

9.09% 

3 

High  school 
graduate/diploma 

15 

26.32% 

6 

17.14% 

9 

40.91% 

4 

Some  college,  no  degree 

19 

33.33% 

11 

31 .43% 

10 

45.45% 

5 

Associate's  degree  or 
other  2-year  degree 

6 

10.53% 

5 

14.29% 

1 

4.55% 

6 

Bachelor's  degree 

12 

21 .05% 

12 

34.29% 

0 

0.00% 

7 

Master's  degree 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

8 

Doctoral  or  professional 
degree 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

Q6 

What  is  your  current 
paygrade? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

E2-E3 

7 

12.28% 

0 

0.00% 

7 

31 .82% 

2 

E4-E6 

21 

36.84% 

7 

20.00% 

14 

63.64% 

3 

E7-E9 

16 

28.07% 

15 

42.86% 

1 

4.55% 

4 

W1-W4 

0 

0.00% 

0 

0.00% 

0 

0.00% 

5 

01-03 

7 

12.28% 

7 

20.00% 

0 

0.00% 

6 

04-06 

6 

10.53% 

6 

17.14% 

0 

0.00% 

Q7 

How  long  have  you 
been  in  the  Navy? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

<1 

2 

3.51% 

0 

0.00% 

1 

4.55% 

2 

1  -  5  yrs 

18 

31 .58% 

5 

14.29% 

13 

59.09% 

3 

5  -  10  yrs 

5 

8.77% 

3 

8.57% 

2 

9.09% 

4 

10  - 15  yrs 

16 

28.07% 

12 

34.29% 

1 

4.55% 

5 

>15  yrs 

16 

28.07% 

15 

42.86% 

1 

4.55% 

A-2 


Q8 

How  often  do  you  use  a 
computer  at  work? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Never 

1 

1 .75% 

0 

0.00% 

1 

4.55% 

2 

Sometimes,  but  not  every 
day 

7 

12.28% 

1 

2.86% 

6 

27.27% 

3 

Every  day,  but  not  all  day 

24 

42.11% 

16 

45.71% 

8 

36.36% 

4 

Most  of  my  work  is  on  a 
computer 

25 

43.86% 

18 

51.43% 

7 

31 .82% 

Q9 

How  often  do  you  use  a 
computer  at  home? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Never 

3 

5.26% 

1 

2.86% 

2 

9.09% 

2 

Sometimes,  but  not  every 
day 

33 

57.89% 

22 

62.86% 

11 

50.00% 

3 

Every  day,  but  not  all  day 

21 

36.84% 

12 

34.29% 

9 

40.91% 

Q10 

1  have  a  clear 
understanding  of  the 
present  EVAL/FITREP 
system. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

15 

26.32% 

13 

37.14% 

2 

9.09% 

2 

Agree 

34 

59.65% 

19 

54.29% 

15 

68.18% 

3 

Neither  agree  nor 
disagree 

3 

5.26% 

2 

5.71% 

1 

4.55% 

4 

Disagree 

3 

5.26% 

1 

2.86% 

2 

9.09% 

5 

Strongly  disagree 

2 

3.51% 

0 

0.00% 

2 

9.09% 

Q11 

My  last  EVAL/FITREP 
was  fair/accurate. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

15 

26.32% 

12 

34.29% 

3 

13.64% 

2 

Agree 

34 

59.65% 

21 

60.00% 

13 

59.09% 

3 

Neither  agree  nor 
disagree 

7 

12.28% 

1 

2.86% 

6 

27.27% 

4 

Disagree 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

A-3 


Q12 

My  last  EVAL/FITREP 
was  conducted  in  a 
timely  matter. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

12 

21 .05% 

10 

28.57% 

2 

9.09% 

2 

Agree 

37 

64.91% 

24 

68.57% 

13 

59.09% 

3 

Neither  agree  nor 
disagree 

4 

7.02% 

0 

0.00% 

4 

18.18% 

4 

Disagree 

4 

7.02% 

1 

2.86% 

3 

13.64% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

Q13 

1  was  able  to  submit  my 
own  input  at  my  last 
EVAL/FITREP. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

15 

26.32% 

14 

40.00% 

1 

4.55% 

2 

Agree 

36 

63.16% 

21 

60.00% 

15 

68.18% 

3 

Neither  agree  nor 
disagree 

4 

7.02% 

0 

0.00% 

4 

18.18% 

4 

Disagree 

2 

3.51% 

0 

0.00% 

2 

9.09% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

Q14 

My  last 

advancement/promotion 
recommendation  was 
fair/accurate. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

17 

29.82% 

15 

42.86% 

2 

9.09% 

2 

Agree 

35 

61 .40% 

19 

54.29% 

16 

72.73% 

3 

Neither  agree  nor 
disagree 

3 

5.26% 

1 

2.86% 

2 

9.09% 

4 

Disagree 

2 

3.51% 

0 

0.00% 

2 

9.09% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

A-4 


Q15 

1  am  satisfied  with  the 
present  Navy 
EVAL/FITREP  system. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

5 

8.77% 

4 

1 1 .43% 

1 

4.55% 

2 

Agree 

26 

45.61% 

17 

48.57% 

9 

40.91% 

3 

Neither  agree  nor 
disagree 

16 

28.07% 

9 

25.71% 

7 

31 .82% 

4 

Disagree 

10 

17.54% 

5 

14.29% 

5 

22.73% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

Q16 

The  most  qualified  and 
deserving  Sailors  score 
the  highest  on  ther 
EVALs/FITREPs. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

5 

8.77% 

3 

8.57% 

2 

9.09% 

2 

Agree 

23 

40.35% 

16 

45.71% 

7 

31 .82% 

3 

Neither  agree  nor 
disagree 

12 

21 .05% 

9 

25.71% 

3 

13.64% 

4 

Disagree 

15 

26.32% 

7 

20.00% 

8 

36.36% 

5 

Strongly  disagree 

2 

3.51% 

0 

0.00% 

2 

9.09% 

Q17 

Have  you  ever  used  a 
PeopleSoft  software 
tool  or  interface? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Yes 

2 

3.51% 

2 

5.71% 

0 

0.00% 

2 

No 

37 

64.91% 

19 

54.29% 

17 

77.27% 

3 

Don't  Know/Not  Sure 

18 

31 .58% 

13 

37.14% 

5 

22.73% 

A-5 


Q18 

How  easy  or  difficult  do 
you  think  it  will  be  to 
use  this  test  version  of 
the  performance 
management  system?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  difficult 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

2 

Somewhat  difficult 

13 

22.81% 

7 

20.00% 

6 

27.27% 

3 

Neither  easy  nor  difficult 

26 

45.61% 

17 

48.57% 

9 

40.91% 

4 

Somewhat  easy 

13 

22.81% 

7 

20.00% 

6 

27.27% 

5 

Very  easy 

0 

0.00% 

0 

0.00% 

0 

0.00% 

*Four  participants  did  not  feel  comfortable  answering  this  question. 


Q19 

How  efficient  or 
inefficient  do  you  think 
the  performance 
management  system 
will  be?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  inefficient 

2 

3.51% 

1 

2.86% 

1 

4.55% 

2 

Somewhat  inefficient 

8 

14.04% 

3 

8.57% 

5 

22.73% 

3 

Neither  efficient  nor 
inefficient 

22 

38.60% 

14 

40.00% 

8 

36.36% 

4 

Somewhat  efficient 

21 

36.84% 

14 

40.00% 

7 

31 .82% 

5 

Very  efficient 

0 

0.00% 

0 

0.00% 

0 

0.00% 

*Four  participants  did  not  feel  comfortable  answering  this  question. 
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Post- Test  Survey 

Fifty-five  participants  completed  the  post-test  survey. 


Q1 

How  comfortable  or 
uncomfortable  did  you 
feel  performing  the 
tasks  in  the  test? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  uncomfortable 

4 

7.02% 

0 

0.00% 

4 

18.18% 

2 

Somewhat  uncomfortable 

19 

33.33% 

8 

22.86% 

11 

50.00% 

3 

Neither  comfortable  nor 
uncomfortable 

5 

8.77% 

3 

8.57% 

2 

9.09% 

4 

Somewhat  comfortable 

19 

33.33% 

17 

48.57% 

2 

9.09% 

5 

Very  comfortable 

8 

14.04% 

6 

17.14% 

2 

9.09% 

Q2 

How  certain  or 
uncertain  are  you  that 
you  completed  the 
tasks  successfully? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  uncertain 

5 

8.77% 

1 

2.86% 

4 

18.18% 

2 

Somewhat  uncertain 

13 

22.81% 

4 

1 1 .43% 

9 

40.91% 

3 

Neither  certain  nor 
uncertain 

4 

7.02% 

3 

8.57% 

1 

4.55% 

4 

Somewhat  certain 

23 

40.35% 

17 

48.57% 

6 

27.27% 

5 

Very  certain 

10 

17.54% 

9 

25.71% 

1 

4.55% 

Q3 

Compared  to  other 
similar  software  you 
have  used,  how  would 
you  rate  this 
performance 
management  system  in 
terms  of  ease  of  use?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Much  more  complicated 

1 

1 .75% 

0 

0.00% 

1 

4.55% 

2 

Slightly  more  complicated 

15 

26.32% 

6 

17.14% 

9 

40.91% 

3 

The  same 

11 

19.30% 

7 

20.00% 

4 

18.18% 

4 

Slightly  less  complicated 

22 

38.60% 

17 

48.57% 

5 

22.73% 

5 

Much  less  complicated 

5 

8.77% 

3 

8.57% 

2 

9.09% 

*One  participant  did  not  answer  this  question. 
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Q4 

Overall,  how  easy  or 
difficult  was  the  system 
to  use? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  difficult  to  use 

1 

1 .75% 

1 

2.86% 

0 

0.00% 

2 

Somewhat  difficult  to  use 

11 

19.30% 

4 

1 1 .43% 

7 

31 .82% 

3 

Neither  difficult  nor  easy 
to  use 

10 

17.54% 

7 

20.00% 

3 

13.64% 

4 

Somewhat  easy  to  use 

28 

49.12% 

19 

54.29% 

9 

40.91% 

5 

Very  easy  to  use? 

5 

8.77% 

3 

8.57% 

2 

9.09% 

Q5 

Overall,  how  easy  or 
difficult  was  the  system 
to  understand?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  difficult  to 
understand 

0 

0.00% 

0 

0.00% 

0 

0.00% 

2 

Somewhat  difficult  to 
understand 

15 

26.32% 

9 

25.71% 

6 

27.27% 

3 

Neither  difficult  nor  easy 
to  understand 

6 

10.53% 

3 

8.57% 

3 

13.64% 

4 

Somewhat  easy  to 
understand 

24 

42.11% 

14 

40.00% 

10 

45.45% 

5 

Very  easy  to  understand 

6 

10.53% 

8 

22.86% 

2 

9.09% 

*Four  participants  did  not  answer  this  question. 


Q6 

Overall,  how 
professional  or 
unprofessional  did  the 
system  appear?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  unprofessional 

2 

3.51% 

1 

2.86% 

1 

4.55% 

2 

Somewhat  unprofessional 

0 

0.00% 

0 

0.00% 

1 

4.55% 

3 

Neither  professional  nor 
unprofessional 

4 

7.02% 

1 

2.86% 

3 

13.64% 

4 

Somewhat  professional 

24 

42.11% 

17 

48.57% 

7 

31 .82% 

5 

Very  professional 

24 

42.11% 

15 

42.86% 

9 

40.91% 

*One  participant  did  not  answer  this  question. 
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Q7 

Overall,  how  efficient  or 
inefficient  was  the 
system? 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  inefficient 

4 

7.02% 

3 

8.57% 

1 

4.55% 

2 

Somewhat  inefficient 

9 

15.79% 

7 

20.00% 

2 

9.09% 

3 

Neither  efficient  nor 
inefficient 

11 

19.30% 

7 

20.00% 

4 

18.18% 

4 

Somewhat  efficient 

20 

35.09% 

12 

34.29% 

8 

36.36% 

5 

Very  efficient 

11 

19.30% 

5 

14.29% 

6 

27.27% 

Q8 

Overall,  as  you  worked 
through  the  tasks,  did 
the  product  become  . . . 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Much  harder  to  use 

0 

0.00% 

0 

0.00% 

0 

0.00% 

2 

Somewhat  harder  to  use 

3 

5.26% 

3 

8.57% 

0 

0.00% 

3 

Neither  harder  nor  easier 
to  use 

7 

12.28% 

2 

5.71% 

0 

0.00% 

4 

Somewhat  easier  to  use 

31 

54.39% 

22 

62.86% 

9 

40.91% 

5 

Much  easier  to  use 

14 

24.56% 

7 

20.00% 

7 

31 .82% 

Q9 

Overall,  how  effective  or 
ineffective  do  you  think 
the  performance 
management  system 
will  be  as  a  career 
development  and  career 
planning  tool?* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Very  ineffective 

2 

3.51% 

1 

2.86% 

1 

4.55% 

2 

Somewhat  ineffective 

6 

10.53% 

3 

8.57% 

3 

13.64% 

3 

Neither  effective  nor 
ineffective 

8 

14.04% 

5 

14.29% 

3 

13.64% 

4 

Somewhat  effective 

27 

29.82% 

19 

54.29% 

8 

36.36% 

5 

Very  effective 

11 

19.30% 

5 

14.29% 

6 

27.27% 

*One  participant  did  not  answer  this  question. 
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Q10 

1  have  a  clear 
understanding  of  the 
present  EVAL/FITREP 
system. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

3 

5.26% 

1 

2.86% 

2 

9.09% 

2 

Agree 

25 

43.86% 

15 

42.86% 

10 

45.45% 

3 

Neither  agree  nor 
disagree 

11 

19.30% 

6 

17.14% 

5 

22.73% 

4 

Disagree 

11 

19.30% 

8 

22.86% 

3 

13.64% 

5 

Strongly  disagree 

5 

8.77% 

4 

1 1 .43% 

1 

4.55% 

Q11 

My  last  EVAL/FITREP 
was  fair/accurate. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

6 

10.53% 

2 

5.71% 

4 

18.18% 

2 

Agree 

33 

57.89% 

21 

60.00% 

12 

54.55% 

3 

Neither  agree  nor 
disagree 

14 

24.56% 

10 

28.57% 

4 

18.18% 

4 

Disagree 

2 

3.51% 

1 

2.86% 

1 

4.55% 

5 

Strongly  disagree 

0 

0.00% 

0 

0.00% 

0 

0.00% 

Q12 

My  last  EVAL/FITREP 
was  conducted  in  a 
timely  matter. 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

6 

10.53% 

5 

14.29% 

5 

22.73% 

2 

Agree 

29 

50.88% 

16 

45.71% 

13 

59.09% 

3 

Neither  agree  nor 
disagree 

16 

28.07% 

10 

28.57% 

6 

27.27% 

4 

Disagree 

2 

3.51% 

1 

2.86% 

1 

4.55% 

5 

Strongly  disagree 

2 

3.51% 

2 

5.71% 

0 

0.00% 

A-10 


Q13 

1  am  satisfied  with  the 
present  Navy 
EVAL/FITREP  system.* 

Overall 

% 

Supervisors 

% 

Non- 

Supervisors 

% 

1 

Strongly  agree 

4 

7.02% 

2 

5.71% 

2 

9.09% 

2 

Agree 

28 

49.12% 

15 

42.86% 

13 

59.09% 

3 

Neither  agree  nor 
disagree 

14 

24.56% 

10 

28.57% 

4 

18.18% 

4 

Disagree 

5 

8.77% 

4 

1 1 .43% 

1 

4.55% 

5 

Strongly  disagree 

3 

5.26% 

2 

5.71% 

1 

4.55% 

*One  participant  did  not  answer  this  question. 
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Appendix  B 

Categorical  Usability  Data 


HPFD  Session 


Task  1  -  Complete  the  CBT  Tutorial 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  does  not  follow  screen 
instructions 

125 

17.8 

170 

24.2 

Tutorial  button  error 

16 

5.33 

42 

6 

User  asks  for  help 

3 

1.5 

1 

1 

Navigational  error 

1 

1 

0 

0 

Task  2  -  Log  into  NSIPS 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  not  able  to  set  new 
password. 

21 

3 

16 

2.28 

User  asks  for  help 

7 

1.16 

11 

1.57 

System  or  server  error 

7 

2.33 

7 

1.4 

User  refers  to  information  sheet 

3 

3 

1 

1 

Navigational  error 

2 

2 

0 

0 

User  is  timed  out 

1 

1 

0 

0 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

1 

1 

0 

0 

Task  3  -  Open  the  HPFD  document. 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Navigational  error 

25 

3.57 

19 

2.71 

User  refers  to  information  sheet 

21 

3 

14 

2 

User  asks  for  help 

8 

1.14 

7 

1.4 

System  or  server  error 

6 

1 

7 

1.16 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

5 

1.66 

1 

1 

User  is  timed  out 

0 

0 

2 

1 

User  searches  outside  HPFD 
session 

1 

1 

1 

1 

General  button  error 

1 

1 

0 

0 
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Task  4  -  Complete  the  HPFD  document 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  refers  to  information  sheet 

22 

3.14 

6 

2 

User  asks  for  help 

11 

1.83 

9 

1.8 

General  button  error 

6 

1.2 

2 

1 

User  is  timed  out 

4 

1 

5 

2.5 

Navigational  error 

7 

2.33 

1 

1 

System  or  server  error 

2 

1 

0 

0 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

3 

1.5 

0 

0 

User  searches  outside  HPFD 
session 

2 

2 

1 

1 

Task  5  -  Check  Spelling 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  timed  out 

10 

1.42 

13 

1.85 

System  or  server  error 

8 

4 

0 

0 

Spell  check  button  error 

5 

1 

4 

1.33 

User  asks  for  help 

4 

1 

2 

1 

User  refers  to  information  sheet 

5 

1 

User  searches  outside  HPFD 
session 

4 

1 

0 

0 

Navigational  error 

2 

2 

0 

0 

Task  6  -  Find  the  “Target  Behaviors”  description 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  timed  out 

8 

1.14 

9 

1.28 

User  refers  to  information  sheet 

5 

1 

1 

1 

General  button  error 

3 

1 

1 

1 

System  or  server  error 

2 

1 

0 

0 

User  asks  for  help 

3 

1 

1 

1 

Navigational  error 

0 

0 

1 

1 
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Task  7  -  Change  ratings  and  cut  and  paste  comments. 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

General  button  error 

0 

0 

6 

6 

User  asks  for  help 

3 

1.5 

2 

2 

User  refers  to  information  sheet 

3 

1 

1 

1 

User  searches  outside  HPFD 
session 

3 

1.5 

1 

1 

System  or  server  error 

1 

1 

0 

0 

User  is  timed  out 

1 

1 

0 

0 

Task  8  -  Collapse  all  sections  of  the  document 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  timed  out 

11 

1.57 

8 

1.14 

Navigational  error 

2 

1 

1 

1 

User  refers  to  information  sheet 

2 

1 

1 

1 

User  asks  for  help 

3 

1.5 

0 

0 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

3 

3 

0 

0 

General  button  error 

1 

1 

2 

2 

System  or  server  error 

1 

1 

1 

1 

User  searches  outside  HPFD 
session 

1 

1 

0 

0 

Task  9  -  Submit  the  HPFD  document 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

User  is  timed  out 

6 

1 

7 

1 

Navigational  error 

0 

0 

11 

11 

User  refers  to  information  sheet 

6 

1 

2 

1 

User  asks  for  help 

1 

1 

5 

1.25 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

1 

1 

4 

4 

General  button  error 

0 

0 

1 

1 

System  or  server  error 

1 

1 

2 

1 

User  searches  outside  HPFD 
session 

1 

1 

0 

0 

B-3 


Task  10  -  Enter  a  performance  note. 


Problem  Category 

Non-Supervisors 

Supervisors 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Total  Number  of 
Incidents  Across 
Sessions 

Average 
Number  of 
Incidents  Per 
Session 

Navigational  error 

16 

2.28 

9 

2.25 

User  refers  to  information  sheet 

12 

1.71 

1 

1 

User  is  timed  out 

4 

1 

2 

1 

User  asks  for  help 

3 

1 

1 

1 

User  retries  action  because  the 
system  did  not  react  the  first 
time. 

3 

1.5 

9 

4.5 

General  button  error 

0 

0 

2 

1 

System  or  server  error 

3 

1 
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ePerformance  Appraisal  Session 


Task  11  -  Log  into  NSIPS  using  the  ePerformance  test  account 


Supervisors 

Problem  Category 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  not  able  to  set  new  password. 

2 

2 

Task  12  -  Open  the  Annual  Performance  Appraisal  1  document 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

Navigational  error 

17 

5.66 

User  asks  for  help 

1 

1 

Task  13  -  Complete  the  Annual  Performance  Appraisal  1  document 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

System  or  server  error 

4 

1 

User  is  timed  out 

2 

2 

User  asks  for  help 

1 

1 

User  retries  action  because  the  system  did  not 
react  the  first  time 

1 

1 

Task  14  -  Check  the  ratings  descriptions  for  one  dimension. 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  timed  out 

1 

1 

User  refers  to  information  sheet 

1 

1 

Navigational  error 

1 

1 

Task  15  -  Check  spelling 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  timed  out 

2 

1 

Navigational  error 

1 

1 

Task  16  -  Check  language 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  timed  out 

5 

1 
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Task  17  -  Calculate  ratings 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  timed  out 

3 

1 

User  asks  for  help 

1 

1 

Task  18  -  Submit  the  Annual  Performance  Appraisal  1  document 


Problem  Category 

Supervisors 

Total  Number  of  Incidents 
Across  Sessions 

Average  Number  of 
Incidents  Per  Session 

User  is  timed  out 

3 

1.5 

Complete  button  error 

1 

1 

User  retries  action  because  the  system  did  not 
react  the  first  time 

1 

1 

User  refers  to  information  sheet 

1 

1 
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Appendix  C 

Supervisory  Personnel: 

Main  Content  Themes  and  Comments 


II. 


Starting  with  the  briefing  that  described  the  new  HPFD  and  ePerformance  plan.  This 
was  designed  to  give  you  a  good  overview  of  the  new  system  and  what  features 
changed  from  our  previous  performance  appraisal  system. 


a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Clarity 


Quotes 

Clear-cut  instruction/overview.  Supervisors  need  to  know  much  more  about 
why  we  are  changing  to  a  new  system  and  what  the  new  features  are  than 
non-supervisors. 

Relevant  and  critical  that  all  personnel  receive/have  access  to  the  overview. 
Most  in  group  agree  that  all  personnel  should  receive  more  background  on 
how  and  why  a  new  system  is  necessary. 

Brief  hit  the  highlights  such  that  I  was  able  to  comprehend  the  system  - 
briefing  should  provide  basic  information  about  “why  I  am  here” 

As  an  introduction,  the  overview  served  its  purpose 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Forced 

Distribution 

Discussion 


Quotes 

Forced  distribution  is  good,  but  it  could  be  better  -  seniority  sometimes 
plays  a  bigger  role  in  “FD”  -  the  way  to  fix  this  is  to  give  the  “EP”  to  the 
guy  who  deserves  it  most,  but  realistically  it  is  the  guy  who  was  there 
longest... ”FD”  is  a  motivating  tool  for  my  Sailors  -  competition  is  good  to 
inspire  better  performance  -  when  you  walk  onboard,  you  know  who  the 
“alpha  dog”  is  and  who  you  need  to  beat  -  we  will  have  to  overcome  the 
current  mind-set  and  become  accustomed  to  using/interacting  with  the  new 
system  and  it’s  way  of  recognizing  behavior/performance 


Wording 
problems  - 
clarity  and 
military  feel 


Process 

questions 


“FD”  is  good  and  bad  -  people  fall  victim  to  the  “new  guy  on  the  block” 
syndrome 

Did  not  see  a  “promotion  recommendation”  block 

If  you  are  trying  to  promote  a  person  -  you  need  to  be  able  to  say  that  you 
are  trying  to  promote  a  person 

Brief  had  a  non-military  feel  which  made  it  hard  to  understand 
Overview  needs  to  go  into  more  detail 

Needs  more  background  information  about  5VM  -  if  it  is  to  be  used  as  a 
manual/instruction 

Need  detailed  instruction  to  understand  why  you  are  doing  something 
F  -  some  terminology  was  unclear 

Brief  did  not  explain  what  was  broken  about  the  old  system  -  what  are  we 
trying  to  fix? 

How  are  we  going  to  integrate  the  new  system  with  the  old  system  records? 
Supervisors  need  to  know  about  the  new  system,  but  they  need  to  know 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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m.  Now  we’ll  talk  about  training  and  reference  materials.  These  materials  were  designed 
to  help  familiarize  you  with  the  system  and  get  you  to  the  documents  necessary  to 
conduct  your  HPFD  and  ePerformance  session  [SOME  PARTICIPANTS  WILL 
RECEIVE  A  HAND-CARD,  OTHERS  WILL  COMPLETE  THE  CBT] 

a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 

Quick  Reference  Guide 

Theme  Quotes 

Usefulness  1  liked  the  QRG 

The  QRG  was  very  useful  -  most  personnel  will  need  this  type  of  guidance 
Need  a  “quick  reference”  -  this  is  critical 

Language,  talent,  experience  barriers  exist  that  will  keep  some  personnel 
from  effectively  performing  an  HPFD/ePerformance  session  or  complete  the 
“paperwork”  -  some  people  write  better  than  others.  The  Sailors  with 
limited  time  and  experience  would  benefit  from  a  “Quick  Reference  Guide” 
-  “We  ‘go  by  the  gouge’  -  that’s  the  way  we  do  most  of  our  work  out  here.” 

Cheat  sheet  was  used  by  3  of  5  participants  -  recommended  labeling  it  as  a 
“Quick  Reference  Guide”. 

It  was  clear,  but  aimed  toward  computer  illiterate  user 

I  needed  the  QRG  -  I  would  have  been  lost  without  it. 

Kids  are  computer  literate  and  are  more  likely  to  understand  the  program 
quickly 

QRG  is  easy  to  understand 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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Computer-based  Training: 


Theme 

General 

Satisfaction 


Quotes 

CBT  made  the  task  simpler  -  good  tool. 

Sorting  tasks  in  Q&A  is  good  -  I  liked  how  the  numbers  told  you  where  to 
look 

CBT  flowed  well  -  The  part  where  there  were  equations  was  hard  to  follow 

CBT  is  relatively  straight-forward  and  helped  me  understand  how  the  system 
works 

CBT  was  not  bad,  but  there  needs  to  be  some  additional  refinement  -  CBT  is 
helpful  since  I  have  never  seen  HPFD/ePerformance  system  before 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 

Quick  Reference  Guide: 


Theme 

Availability  & 
Accessibility 

Quotes 

Did  not  know  QRG  was  available 

Quick  reference  was  does  not  contain  “in-line  instruction” 

Presentation  of 
Information  - 
Relevance 

It  would  have  been  helpful  to  highlight  functions  and  critical  areas  of  the 
program 

Did  not  initially  know  that  quick  reference  guide  was  relevant 

Miscellaneous 

Getting  to  the  document  was  the  hard  part  -  modifying  the  document  was 
easy 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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Computer-based  Training: 


Theme 
Clarity  of 
Information 


Relevance 

Simplicity  of 

Learning 

Process/Didn’t 

Relay  Document 

Navigation 

Information 


Quotes 

Acronyms  were  hard  to  understand  -  were  not  given  clear  definitions 

Did  not  know  where  to  look  on  the  screen  -  too  much  to  look  at. .  .member 
will  have  to  filter  too  much  info 

CBT  should  be  “dumbed”  down.  I  didn’t  know  what  to  focus  on. 

Acronyms  were  too  vanilla  -  needs  more  of  a  military  feel  to  the  tutorial. 

The  term  supervisor/employee  were  confusing.  We  should  use  terminology 
that  we  currently  use  like  Sailor,  rater,  senior  rater,  etc. 

CBT  did  not  cover  big  picture  to  explain  the  "why"  &  "when"  specific 
functions  are  performed  -  need  "good  process  overview" 

Navigation  functionality  within  CBT  is  not  clear  - 

Learning  objectives  were  not  obvious.  It  provided  a  familiarity  rather  than 
learning  how  to  use  the  system. 

Not  easy  to  discriminate  between  "learning  objective"  from  "CBT 
framework"  -  what  do  I  need  to  focus  on  in  a  given  screen  -  text  too  small 
R&T:  CBT  did  not  help  me  find  the  documents  -  it  only  helped  filling  out 
the  documents 

Not  "idiot  proof'  -  inexperienced  users  may  need  specific  steps 

CBT  is  tedious  (VERY  boring)  -  I  actually  fell  asleep  during  the  CBT 
session  -  what  is  main  focus  in  each  frame  of  CBT  -  do  I  look  at  the  study 
window  (top)  or  do  I  look  at  the  notes  window  (bottom)  -  hand-out  (QRG) 
would  be  helpful 

First,  for  the  first  3  slides,  I  focused  on  the  image  of  the  computer  screen. 
When  I  realized  that  nothing  changed  on  those  three  screens,  I  noticed  the 
text  box  on  the  bottom.  It  was  really  frustrating. 

Move  text  box  (bottom)  to  top  -  swap  location  of  example  with  location  of 
objective  text 

It  looked  detail,  but  it  lacked  simplicity  - 

Nothing  worked  as  expected  -  scroll-bars,  links,  etc.  display  not  react 
because  it  was  static. . . 

Sailors  work  from  checklists  -  they’re  used  to  that.  The  training  should  be 
set  up  as  a  checklist  to  work  people  through  the  training. 

Include  some  real-world  examples  in  the  training  that  people  can  relate  to. 

Need  integrity  in  training  -  accessibility  will  be  critical  -  suggest  a  stand¬ 
alone  system 
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Miscellaneous 


Daily  progress  updates  should  be  done  “face  to  face” 


Recommended  a  classroom,  interactive  training  rather  than  CBT.  E- 
Learning  puts  shipboard  Sailors  at  a  competitive  disadvantage  because  they 
don’t  have  the  opportunity  to  check  email  daily,  access  computer  terminals 
to  complete  work,  and  use  the  internet  for  training  and  professional 
development.  “Being  at  sea  is  what  the  Navy  is  all  about.  Initiatives  need  to 
be  designed  for  the  at-sea  people  first  and  tested  at  sea  and  then  move  it  to 
shore  commands.” 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 

KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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IV.  Starting  with  logging  onto  the  system.  You  needed  to  do  several  tasks  -  enter  a 

username,  password,  and  select  the  Navy  Performance  Management  and  Appraisal 
system. 

a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Ease  of  Login 
When 

Functioning 


Quotes 

I  thought  it  was  easy. 

When  the  system  is  working  properly  -  login  is  very  easy 
3  of  5  who  were  able  to  log  in  did  so  with  little  difficulty. 
Login  was  smooth 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Login  Failures 
due  to  NSIPS 
Problems 

Quotes 

The  connection  was  very  slow  and  we  have  a  T3  connection  here. 

Very  slow  system  access  -  it  took  too  long  to  login.  I  would  have  given  up 
and  tried  another  time  if  this  were  for  real. 

Supervisor  password  not  working 

System  kept  dropping  -  4  login  attempts  finally  enabled  me  to  access  my 
HPFD  document 

Login  failed  -  required  workaround 

Got  kicked  out  of  the  system  and  was  required  to  login  again  -  it  ended  up 
working,  but  the  “hiccup”  was  disconcerting 

Need  the  faster  internet  connection  -  connectivity  is  going  to  be  a  show 
stopper  -  internet  connection  is  unreliable  underway 

Unclear 

Password 

Requirements 

Why  is  user  name  case-sensitive? 

Why  is  my  account  name  “case  sensitive”?  -  I  stumbled  around  the  menu 

Case  sensitivity  of  user  name  and  password  was  frustrating  since  there  is  no 
indication 

User  name  should  NOT  be  case  sensitive 

The  rules  for  password  generation  should  be  published 

Was  not  sure  that  password  was  successfully  changed 

Miscellaneous 

No  problems  logging  on. 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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V. 


Now  let’s  discuss  selecting  and  opening  your  document.  Here  you  are  asked  to 
identify  the  appropriate  document,  select,  and  open  your  document  awaiting  action. 


a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Document  Easy 
to  Find  and 
Select 


Quotes 

Having  the  document  link  highlighted  made  the  document  easy  to  find. 
Document  readily  available 

Easy  to  operate  system  once  you  realize  how  everything  works  and  where  to 
locate  information  (it  takes  about  10  minutes  to  start  becoming  comfortable) 


Easy  to  navigate 

It  was  intuitive  for  me  to  look  around  like  I  look  around  in  other  new 
software  programs  I  use.  I  work  with  computers  all  day  long,  so  I’m  used  to 
doing  that. 


Ease  of  Selecting 
Document 
Facilitated  by 
Quick  Reference 
Guide 


3  of  5  used  the  QRG  to  find  and  select  their  document 
I  used  the  QRG  and  whipped  right  through  the  process 
Using  the  QRG  made  sense  of  the  system 
I  used  the  QRG  to  help  me  get  to  my  document 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Quotes 

Where  do  I  go  to  find  a  document? 

Did  not  know  what  kind  of  document  I  was  supposed  to  be  looking  for 

What  does  “my  document”  mean  -  had  to  review  someone  else’s  document 

What  are  all  of  the  various  menu  option  (links)  there  for  -  the  link  to  pull 
up  my  document  is  not  obvious 

Putting  the  term  “document”  together  with  “evaluation”  is  difficult 

System  navigation  is  unclear  -  where  do  I  start  once  I  get  logged  in?  - 
Navigation  is  not  intuitive  and  the  system  was  not  explained  well  in  the 
CBT 

A:  once  logged  in  -  how  do  I  find  the  document?  Starting  off,  I  was  lost.  I 
looked  for  a  “help  button”  but  it  wasn’t  there.  The  menu  was  not  intuitive. 

There  were  too  many  possible  areas  for  me  to  get  lost  in.  I’d  recommend 
displaying  only  the  options  you  are  authorized  to  use. 

Left  side  of  the  menu  needs  to  be  clearer  -  too  many  choices. 

Functions  not  well  defined.  Reorganize  critical  tasks/functions  and  make 
more  user-friendly. 

Finding  the  document  to  open  was  difficult  for  all  three.  When  asked  what 
strategy  users  used  to  find  and  open  documents,  all  said  that  they  hunted 
around,  clicked  on  links,  found  what  they  had  access  to,  and  looked  for  the 
documents. 

I  think  there  should  be  quick  links  that  bring  you  either  into  the  HPFD  part 
or  the  ePerformance  part. 

Did  not  know  where  to  locate  document 

Would  probably  use  a  search  tool  to  find  documents 

Too  many  button  clicks  required  to  find  the  document.  The  opening  page 
should  take  you  right  to  the  list  of  documents. 

The  menu  system  did  not  display  an  obvious  “place-holder”  when  you 
expanded  a  menu  option 

First  time  through  -  function  is  not  user  friendly 

Sub-menus  are  not  clear  and  do  not  help  me  understand  what  it  is  I  supposed 
to  do  -  the  quick  reference  guide  cleared  things  up 

Page  Loading  Access  is  too  slow  -  page  loads  too  slowly  -  people  likely  to  try  reloading 
rather  than  wait 


Theme 
Trouble 
Identifying  the 
Document 


Trouble 

Navigating  to  the 
Document 
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Terminology  Too  The  menus  does  not  use  military  “flavored”  wording  causing  the  Sailor  to 
Civilian  make  the  mental  jump  to  make  sense  of  the  “civilian”  verbiage. 

Miscellaneous  Task  description  did  not  line  up  with  the  activity  of  the  task  -  required 

assistance  from  the  study  team  to  know  where  to  start  -  menu  titles  were  not 
intuitive. 

Make  sure  the  QRG  is  available 

Search  on  social  or  last  4  of  SSN  rather  than  another  number  that  replaces 
Empl  ID 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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VI.  Let’s  talk  about  completing  (filling  in)  your  HPFD  form.  Let’s  list  the  aspects  of 
completing  the  form  -  filling  in  the  background  information,  completing  the 
performance  dimension,  finalizing  the  form. . . 

a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Quotes 

General  Positive 
Comments 

Expanding  the  document  -  terminology  -  spell  check  -  everything  worked 
well  and  was  easy  to  understand. 

Overall  -  HPFD  document  was  “to  the  point”  in  it’s  implementation  -  the 
“comments”  block  helped  to  balance  out  the  HPFD  rating  -  the  behavioral 
description  was  very  helpful  in  focusing  on  the  performance  -  comments  in 
the  document  should  be  short/to  the  point  and  use  your  “face  to  face”  to 
clarify  your  words 

HPFD  and  ePerformance  felt  different  to  me  (which  is  good) 

There  were  no  roadblocks.  The  form  was  easy  to  complete  and  fill-in  the 
block  with  comments.  It  was  simple  and  efficient.  I  liked  the  spell-check. 

I  liked  the  spell-check 

Examples  of  required  data  and  data  descriptions  are  very  helpful  in 
completing  the  document 

Performance 

Dimensions 

Performance  categories  are  in  tune  with  today’s  Navy 

Liked  the  words  used  in  the  verbal  anchors.  Phrases  like  “Walk  the  talk” 

. .  .those  are  things  we  say  . .  .phrases  I’d  use  in  the  written  text  of  a  FITREP 
or  EVAL. 

Performance  categories  were  well  organized  and  worded  logically 

The  target  behaviors  and  descriptors  were  very  well  worded. 

Where  NAVFIT98  is  vague  -  HPFD  descriptors  across  behaviors  are  too 
verbose  and  have  common  themes  -  behavior  descriptions  need  to  be  more 
direct 

HPFD  Process 

Questions  were  asked  about  the  HPFD  process  . . .  how  frequent,  what  is  the 
gameplan.  Once  described,  [THE  PARTICIPANT]  liked  that  this  system 
and  process  will  facilitate  more  frequent  and  immediate  feedback  that  will 
give  Sailors  the  opportunity  to  make  performance  changes  before  the 
FITREP/EVAL  is  due  to  be  completed. 

I  like  the  email  notification  for  late  mid-term  counseling  sessions.  It  would 
be  even  better  if  your  supervisor  received  an  email  when  these  are  past  due. 
Then  the  supervisor  knows  that  their  work  center  supervisor  needs  to  get 
their  counseling  done  -  accountability. 

Web-based 

Form 

It  was  very  intuitive.  I  liked  the  drop-down  boxes.  It’s  hard  to  mess  up  with 
drop-down  boxes. 
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Easy  to  navigate  from  box  to  box 


Process 

Ownership 


I  liked  the  open-ended  text  box  that  I  can  use  to  help  the  Sailor  improve 
I  liked  the  way  each  section  was  broken  down 
Able  to  select  “rating”  and  make  comments  easily 

I  also  like  the  idea  that  junior  Sailors  are  completing  a  self-assessment  and 
sending  it  to  their  supervisor  to  review  and  provide  feedback.  This  puts 
them  in  the  position  of  having  to  be  responsible  for  their  careers  early  on. 
Many  junior  Sailors  provide  input  and  their  supervisor  writes  their  EVAL. 
Then  when  they  become  a  Second  Class,  they  are  surprised  to  learn  they 
have  to  write  their  own  and  other  people’s  EVALs.  It  also  helps  them  work 
on  their  writing  skills  and  get  feedback  early  on,  making  them  better  writers. 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 
Performance 
Dimensions 
Repetitive  and 
“Too  Civilian” 

Quotes 

Some  behaviors  appear  overlapping  -  when  completing  the  form,  I  found 
myself  thinking  “I’ve  already  answered  this  one  already” 

Although  it  appeared  that  two  sections  were  identical  - 

Trait  descriptions  have  a  civilian  flavor  and  don’t  necessarily  apply  to 
military  culture.  Concerned  that  less  educated  Sailors  will  not  be  able  to 
understand  the  workplace  behaviors  and  the  examples  of  high,  average,  and 
low  performers. 

Rewrite  the  traits  so  they  have  a  military  flavor 

What  type  of  comments  are  appropriate  in  the  behavioral  traits  block?  What 
kind  of  comments  are  expected?  I  would  rather  use  the  HPFD  as  a  type  of 
“brag  sheet”  -  would  like  to  see  system  put  onto  INTRANET  to  work 
around  the  Internet  connectivity  issue 

“Military  bearing”  Sailors  don’t  go  outside  to  play  anymore  -  they  stay 
inside  and  play  computer  games  -  Sailor  cannot  pass  PRT  (this  gets  a  2.0), 
but  shows  up  to  work  wearing  a  crisp  uniform  and  shiny  boots  (this  gets  a 
4.0)  -  there  has  to  be  some  kind  of  compromise  -  what  kind  of  score  do  you 
give  him? 

HPFD  Process 

Where  does  the  document  go  -  how  is  the  document/information  stored  - 
who  sees  it?  Do  I  need  to  hang  onto  a  copy  of  the  document  (just  in  case  I 
need  to  answer  a  Congressional)  -  can  I  hang  onto  a  copy  of  the 
document???  Will  a  Sailor  see  the  HPFD  document  before  I  have  a  chance 
to  have  a  “face  to  face”? 

HPFD  should  be  face-to-face  -  eDocument  is  good  CYA  tool  -  but  what  is 
the  liability? 

I  want  to  do  a  face-to-face  counseling  session  with  Sailors  first,  complete 
the  counseling  document  and  send  it  to  the  Sailor  for  review,  and  they  can 
call  if  they  have  any  problems. 

Will  a  bright  young  Sailor  be  able  to  circumvent  my  opinion  by  writing 
something  up  prior  to  our  “face  to  face”  after  reviewing  his/her  copy  of  my 
HPFD  document? 

Perform  “face  to  face”  prior  to  filling  in  a  HPFD  session  to  “document”  a 
behavioral  trend  -  if  you  take  anything  writing  to  a  “face  to  face”  simply  use 
talking  points  and  then  document  your  comments 

I  would  prefer  to  write  up  the  document  before  having  a  “face  to  face” 

Paper  Document 
Back-up 

Wants  a  “Save  As”  for  a  personal  form  -  a  back-up  copy 

Backup  system  -  what  happens  the  system  fails 
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NSIPS  Problems 


Text  Box 
Limitations 


It  would  be  good  to  have  a  back-up  for  the  system.  You  will  have  a  large 
number  of  people  using  the  system  and  it  may  be  tough  to  gain  access.  I’d 
like  to  have  the  ability  to  “Save  As”  and  save  a  copy  to  my  hard  drive  or  a 
“Print  Version”  so  I  can  work  from  a  hard  copy  for  those  people  who  don’t 
have  computer  access. 

May  explore  having  a  “Print  Version”  that  sends  the  document  holder  a  PDF 
of  the  form  for  them  to  work  from.  Also  have  batch  print  versions  of  forms 
in  one  PDF  so  users  can  back  up  all  documents  at  once. 

I  want  to  have  a  copy  of  the  counseling  document  for  my  files  and  I  want  a 
copy  of  the  counseling  document  to  go  to  the  Sailor’s  new  supervisor  when 
they  transfer.  That  way  Sailors  who  receive  adverse  counseling  won’t  tear 
up  that  counseling  sheet  when  they  transfer. 

Need  a  print  version  of  the  document 

A  printable  form  so  Sailors  can  work  from  a  paper  copy  and  input  their  text 
when  they  have  access  to  a  computer. 

Serves  as  a  record  of  the  session  for  tracking  development,  adverse 
counseling,  and  as  a  back-up  copy. 

It  would  be  helpful  to  have  a  “hardcopy”  of  the  HPFD  output  as  a  working 
copy  when  the  internet  is  down  -  who  sees/maintains  these  documents  and 
how  do  I  transfer  ownership  of  these  documents  to  a  new  supervisor  -  have 
supervisor  hierarchy  accounts  be  position  based  so  that  the  underlying 
hierarchy  is  not  destroyed  when  someone  leaves... 

I  want  a  “hard  copy”  to  back  up  the  “soft  copy” 

Spell-checker  and  help  button  fails  once  the  account  "times  out" 

Spell-check  did  not  work  -  the  page  expired  -  clicked  link  to  “return  to  last 
active  page” 

“Complete”  button  did  not  work 

Must  save  document  if  not  complete  -  or  you  might  lose  your  data 

Dropped  connections  will  cause  personnel  to  “leave  and  come  back”  - 

Pages  froze  up  making  it  difficult  to  work  with  -  could  not  see  output  due  to 
“page  timeout  -  i.e.  Page  not  available  error”) 

Pages  kept  “timing  out”  while  I  was  working  on  the  form  -  if  HPFD 
function  is  only  for  “mid-term  counseling”  maybe  we  should  scrap  that 
concept  and  better  incorporate  (or  modify)  the  ePerformance  function  to 
represent  a  “mid-term”  session  in  a  common  location  -  big  face-to-face 
option  with  a  print  option  so  you  can  take  a  piece  of  paper  to  the  face-to- 
face  session 


No  indication  of  system  limitations  -  need  a  description  of  how  far/how 
much  information  can  be  included  in  the  text  boxes. 
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Limit  the  write-ups  and  focus  on  the  traits  -  people  will  feel  like  they  have 
to  fill  all  the  white  space. 

Limit  the  number  of  lines  or  require  a  certain  number  of  line  of  text  in 
comments  block 


Navigation  & 
Process  Clarity 
Needed 


Could  not  tell  if  HPFD  is  more  of  a  “brag  sheet”  as  compared  to  a  genuine 
“self-assessment”  -  could  not  tell  if  I  was  performing  a  “mid-term 
counseling”  or  a  “performance  evaluation”  -  HPFD  distinction  from 
ePerformance  is  not  clear 


I’m  just  a  dumb  construction  ogre.  Just  show  me  the  tasks  that  I  need  to 
complete  and  what  I  need  to  do  today. 

I  confused  the  HPFD  and  ePerformance  documents 

Conduct  spell-check  on  all  boxes  as  part  of  the  save  function 

“Would  like  to  spell  check  all  at  once 

Make  behavioral  description  more  obvious  -  I  did  not  know  that  I  could 
expand/review  the  descriptions  -  add  separate  button/entry  so  that  I  can 
review  behavioral  traits 


Miscellaneous  Need  accountability  on  counseling  -  you  should  get  paid  if  you  get 

advanced  -  write-up  should  reflect  “what  have  you  done  for  the  Navy 
lately?”  how  do  you  break  out  a  Sailor 

E5  and  below  does  not  have  internet  capability  -  we  setup  a  computer  lab, 
but  only  5  computers  have  internet  access  that  can  be  used  and  my  section 
has  over  100  personnel  that  will  need  to  use  these  computers 
Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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VII.  Now,  let’s  talk  about  completing  (filling  in)  your  ePerformance  form.  Some  of  you 
will  complete  a  document  for  yourselves  as  the  employee  while  others  will  be 
working  on  documents  for  those  who  work  for  you. 

a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Quotes 

Easy  to  Use 

U  ser  friendly 

Very  intuitive  system. 

Form  Design  - 
Text  Boxes  for 
Each 

Performance 
Dimension  & 
Overall 

Summary  Block 

Space  to  make  comments  for  each  trait  is  sufficient  to  make  proper 
comments.  I  like  that  each  trait  has  a  text  box  associated  with  it.  The  trait 
descriptors  are  so  good  I  didn’t  feel  like  I  needed  to  add  text. 

A  separate  comment  block  for  each  behavior  is  ideal 

I  liked  the  overview  box.  This  would  be  a  good  place  if  someone  wanted  to 
send  a  message  to  the  board. 

Forced  distribution  is  not  a  fair  way  of  promoting  a  Sailor  - 

New  system  will  be  more  fair  because  your  qualifications  -  break  down  peer 
comparison  to  platforms 

Text  box  is  a  valuable  “reality  check”  that  will  help  you  better 
visualize/support  a  given  performance  rating.  This  will  help  raters  to  align 
their  comment  with  their  numerical  rating  while  they’re  filling  out  the  form. 

Clarity  of 

Performance 

Dimensions 

Each  behavior  description  is  a  lot  clearer  -  easier  to  rate  people  than 
NAVFIT98.  Descriptions  of  anchors  was  good 

Trait  descriptions  are  broken  down  very  well  -  Descriptions  of  workplace 
behaviors  helps  to  populate  text. 

Organizational  Savvy  is  a  wonderful  behavioral  trait  -  1  liked  the  expanded 
trait  descriptions...  being  able  to  pull  up  additional  descriptive  data  to  help 
you  better  understand  a  given  behavioral  category  -  this  system  will  drive 
you  to  provide  more  feedback  during  the  initial  reporting  period 

Miscellaneous 

5VM  will  fix  the  problem  with  forced  distribution 

Some  participants  did  not  complete  a  performance  appraisal  form  due  to 
technical  problems 

Switch  over  EVERYONE  to  7-point  scale  at  the  same  time  to  be  fair  H  - 1 
like  the  ability  to  hit  “calculate”  rating  and  then  go  back  to  make  sure  your 
numbers  correlate  with  your  comments  before  you  confirm. 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 
Performance 
Dimensions 
Repetitive  and 
“Too  Civilian” 

Quotes 

Rating  descriptor  "does  not  meet  standards"  is  too  negative  -  would  like 
"more  constructive"  verbiage  in  order  to  build  a  Sailor 

“Target  behavior"  does  not  necessarily  apply  to  military  climate  -  they 
sound  more  "politically"  oriented  rather  than  "real  world  -  in  the  trenches" 

Behavioral  descriptions  will  promote  plagiarism  for  those  folks  who  are  not 
creative  writers 

Mission  and  one  other  trait  overlapped  too  much 

Trait  description  were  too  civilianized  -  they  felt  like  “ego  stroking” 
descriptions  and  not  really  relevant  in  a  military  environment 

Would  like  to  see  a  trait  description  for  each  level  of  performance. . .  what 
does  “low  performance”  look  like  as  compared  to  “high  performance” 

ePerformance 

Process 

Concerns 

Lowest  appropriate  supervisory  level  needs  to  perform  and  this  person  needs 
to  know  the  employee  -  drop-down  boxes  and  fill-in  fields  too  impersonal 
Will  there  be  a  block  for  someone  who  does  not  agree  with  their  evaluation? 
There  will  be  a  “I  do  not  plan  to  submit  a  statement”  box,  but  how  would 
they  submit  a  statement? 

Force  the  rater  to  write  a  comment  prior  to  assigning  a  number  rating  -  this 
will  provide  the  rater  with  an  efficient  logical  progression  with  which  to 
create  a  realistic  evaluation 

What  happens  with  detaching  FITREPS? 

How  will  the  new  periodicity  affect  selection  boards  -  would  we  go  to  a 
monthly  board? 

Combining  access  to  “mid-term”  with  “FITREP”  function 

Cultural  Issues  - 
Text  Boxes 

System  has  potential  for  exaggerated  performance  evaluation  -  culturally, 
people  will  want  to  fill-in  the  white-space.  Something  is  going  to  need  to  be 
done  to  keep  people  from  gaming  the  system.  The  white  space  should  be 
restricted  to  examples  of  why  the  person  either  exceeds  or  fails  to  meet 
performance  standards. 

Question  was  asked  the  number  of  characters  per  text  box.  YN 1  stated  it 
was  255  characters  per  box.  Group  thought  that  restricting  the  number  of 
characters  for  each  box  might  prevent  raters  from  getting  too  “wordy  or 
flowery”  in  their  text. 

There  needs  to  be  some  limit  on  the  amount  of  text  for  each  of  the 
workplace  behaviors.  We  already  have  18  lines  and  that  is  about  right  - 
maybe  go  to  25  total  lines.  Comments  need  to  be  concise. 

Limit  number  of  lines  -  18  line  limit  -  only  write  comment  if  you  need  to 
“justify”  an  extremely  high  or  low  rating 
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Cultural  Issues  - 
Promotion 
Recommendatio 
n  &  Forced 
Distribution 


Miscellaneous 


“I  have  two  words  for  you,  ‘Cause’  and  ‘Effect’.  What  did  someone  do  that 
contributed  to  the  ship’s  mission,  helped  out  in  town,  etc.” 

Unlimited  text,  but  don’t  penalize  someone  for  not  entering  text.  Use  a 
“bottom-line”  comment  to  summarize  -  use  key  word  to  indicate  significant 
accomplishments 

Use  concise  writing  style  - 

Forced  distribution  is  necessary  -  justification  comments  should  include 
only  relevant  supporting  information.  We  have  to  have  some  way  of  racking 
and  stacking  people  within  a  command.  That  would  keep  raters  honest. 

How  will  “top  performers”  be  given  the  opportunity  to  take  an  advancement 
test  early? 

All:  What  about  various  recommendations?  (retention,  separation,  thoughts 
for  the  future,  advancement,  seaman  to  admiral,  etc). 

How  do  you  “break  out”  a  Sailor  -  need  a  performance  recommendation  or 
discover  a  different  way  of  “breaking  out”  a  Sailor  -  a  better  way  of 
“breaking  out”  Sailors  must  take  specialties  and  communities  into  account 
(i.e.  you  cannot  lump  all  pilots  into  the  same  group) 

Will  harder  to  distinguish  a  Sailor  from  his/her  peers  -  not  all  Sailors  have 
the  opportunity  to  obtain  your  quals 

Peer  comparison  is  unfair  because  Sailors  are  not  in  the  same  situations 

Human  politics  are  going  to  be  VERY  difficult  to  overcome  -  the  system 
must  override  the  “good  old  boy  network” 

File  "save  as"  -  does  not  trust  career  to  be  saved  on  a  corporate  server 

Also,  this  is  nit-picky,  but  one  of  the  tasks  asked  me  to  cut  and  paste  from 
one  box  to  another.  I  put  my  cursor  over  the  text,  highlighted  it  and  hit  copy. 
It  didn’t  take  because  my  cursor  wasn’t  right  over  the  highlighted  text. 

Instructions  at  top  of  page  were  skimmed/ignored  until  I  became  confused 
and  realized  that  I  needed  some  guidance  in  order  to  complete  the  document 

Perhaps  we  can  “repeat”  the  instructions  at  the  bottom  of  the  page  or  put 
relevant  instructions  into  a  separate  browser  windowpane. 

It  would  be  good  for  this  system  to  fall  under  the  NKO  login.  You  could 
login  to  NKO  and  have  a  link  to  either  the  HPFD  or  the  ePerformance 
section.  Right  now  we  have  so  many  logins  and  passwords,  I  have  to  keep  a 
book  with  me  because  I  have  them  all  written  down  -  NKO,  NMCI, 
NALCOMIS  (Maintenance  Log  system),  and  others.  Your  NKO  login 
should  give  you  access  to  HPFD  and  ePerformance. 

Easy  to  use  -  raters  tend  to  avoid  harsh  language  to  keep  from  "harming"  a 
Sailor  in  front  of  a  board  -  they  don't  know  where  a  report  will  go  and  tend 
to  inflate  the  "rating"  rather  than  being  "honest"  -  training  is  key  to  helping 
Sailors  understand  how  to  effectively  use  the  system  -  push  the  "review" 
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down  to  the  lowest  appropriate  supervisory  level 

HPFD  gives  the  Sailor  (subordinate)  more  ammo  with  which  to  challenge  an 
evaluation 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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VIII.  Let’s  talk  about  wrapping  up  or  closing  your  HPFD  or  performance  appraisal  session. 


a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme  Quotes 

General  Positive  ALL  -  say  as  much  as  you  want  for  HPFD,  but  limit  your  comments  when 
preparing  an  ePerformance  appraisal 

New  system  is  better  in  that  it  does  not  limit  your  evaluation  to  three  or  four 
sample  bullets 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS  KITTYHAWK 
(CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Spell 

Check/Language 

Check 

Quotes 

I  would  like  to  see  a  forced  spell-check. 

“Spell  check”  was  not  easy  to  find  -  put  a  single  button  to  allow  the  rater  to 
run  “spell  check”  prior  to  finalizing  the  document 

Language  check  does  not  do  what  you  expect  it  to  -  I  expected  the  button  to 
check  for  bad  words  or  grammatical  errors  -  I  would  like  the  ability  to 
create  bullets  to  describe  behavior  rather  than  being  forced  to  create  formal 
grammatically  correct  sentences 

Back-up 

Copies/Printable 

Version 

I  need  a  piece  of  paper  -  staring  at  a  screen  is  difficult 

Paper  is  more  comfortable  when  proofing  a  document 

Backup  copy  is  necessary 

I  would  like  to  archive  a  copy  of  a  HPFD  document 

Forced 

Distribution  & 
Promotion 
Summary 
Recommendation 

Allow  rater  to  say  anything  they  want  about  each  behavior  -  put  top  three 
highlights  into  the  overall  summary  block. . .  if  the  summary  block  doesn’t 
“sell”  the  Sailor,  you  can  go  back  to  the  behavior  summary  blocks  for  more 
info. 

The  summary  block  should  be  used  as  a  message  block  to  talk  to  selection 
boards . . . 

Force  the  rater  to  pick  the  three  most  critical  bullets 

Form  Routing 

Routing  options  missing  -  what  do  you  do  with  the  document  when  you  get 
done  filling  it  in? 

Who  will  setup  hierarchy?  Someone  in  "personnel"  (e.g.  personnel  officer) 
should  fulfill  this  duty. 

How  will  changes  in  personnel  duties  affect  the  hierarchy  -  who  will  update 
the  system?  How  easy  will  it  be  to  update  the  system? 

What  is  the  final  disposition  of  your  ePerformance  document?  What  does 
the  final  product  look  like? 

Save  and 

Complete/ 

Forward 

Document 

Function 

Add  the  word  “forward”  to  the  [complete]  button  or  possibly  create  a  pop-up 
confirmation  window  for  those  folks  who  accidentally  click  the  complete 
button.  This  would  help  differentiate  between  closing  the  document  and 
completing  the  document  and  forwarding  it  to  your  supervisor. 

Did  not  read  the  CBT  to  know  that  “hitting  the  [complete]  button  would 
route  the  document  to  the  supervisor” 

Need  a  confirmation  window  to  ensure  you  don’t  accidentally  perform  an 
action  that  you  cannot  undo 
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I  need  “complete”  button  to  be  better  defined  -  what  is  it’s  function? 

Function  of  “complete”  button  is  not  clear  -  what  does  it  do  -  provide  some 
instruction  or  make  the  button’s  function  more  clear 

Text  Boxes  Is  there  a  limit  to  the  number  of  lines  you  can  enter  in  a  block?  What 

weights  are  being  assigned  to  skill  sets/billets/etc 

Give  the  user  a  limited  number  of  input  and  allow  them  to  choose  which 
textbox  to  put  the  text  -  (e.g.  write  your  best  20  lines  of  text  [to  be  shared  by 
all  behavior  boxes]  to  express  the  Sailor’s  accomplishments) 

Distribute  bullets  across  traits. . .  use  your  limit  of  bullets  as  you  see  fit 

Why  limit  the  amount  of  text. .  .write  as  much  as  you  want 

Some  CO’s  are  long-winded  and  require  you  write  more  than  is  necessary  - 
you  must  condense  (limit)  the  amount  of  text  or  else  you  will  fall  back  into 
the  same  situation  we  have  now 

CO’s  say  “white  space”  is  bad  -  you  have  to  fill  up  the  space  -  limit  each 
trait  to  a  couple  of  bullets  instead  of  having  unlimited  input  capability  - 
people  write  to  justify  a  5.0  Sailor  even  though  it  isn’t  required 

Miscellaneous  Complete  button  failed  -  document  “timed-out”  -  make  the  “complete 

button”  function  properly 

Like  saving  a  “draft”  copy  of  these  documents  so  I  can  go  back  later  to 
complete/modify  an  incomplete  performance  document 

On-line  system  is  easy  (for  admin  personnel)  -  not  all  personnel  will  have  a 
PC  available  at  any  given  moment 

Disagreed  with  others  that  online  time  is  limited.  You  think  it  is  hard  to 
come  by,  but  if  your  supervisor  and  his  chief  knew  that  they  were  being 
rated  by  the  number  of  complete  documents  forms  from  their  people  they 
would  arrange  to  make  time  for  you  to  be  online. 

Make  an  “auto  save”  feature  that  saves  a  document  to  your  local 
machine/network  or  allows  you  to  import  a  document  from  your  local 
mac  hine/net  work . 

Program  should  be  loaded  locally  and  connected  to  the  NSIPS  server  to 
upload  your  completed  document  for  routing 

Limit  printed  output  of  one  page 

System  will  ADD  to  our  administrative  burden  -  wording  will  grow  so  you 
keep  up  with  the  Jones’  - 1  don’t  want  to  have  to  justify  every  7.0  rating 

System  is  slow...  pages  load  very  slowly 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS  KITTYHAWK 
(CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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IX.  If  you  had  to  assign  a  grade  to  this  system,  what  grade  would  you  assign? 
(A=Outstanding,  C=Average,  F=Failed) 

a.  What  grade  would  you  assign  to  this  pilot  system?  What  led  you  to  assign  this 
grade? 

b.  What,  if  anything,  could  be  done  to  cause  you  to  raise  your  grade  of  this  system? 


Group  Current  GPA 

NAS  Brunswick  1.70 

Brunswick,  ME 

USS  KITTYHAWK  (CY63)  1 .75 

Yokosuka,  Japan 

Kitsap  Naval  Base  -  Bangor  2.25 

Bangor,  WA 


Theme 

NSIPS/ 

Connectivity 

Problems 


Quotes 

Too  many  problems  - 1  kept  getting  kicked  out  of  the  system 

I’d  grade  this  an  A  for  useability  and  a  D  for  functionality 

Less  waiting  time  -  slow  connection  was  frustrating 

The  concept  is  noble,  but  the  system  was  slow,  it  broke  down,  it  was 
frustrating 

The  system  was  slow,  account  access  was  broken,  not  everything  was 
accessible  - 1  am  not  resistant  to  change,  but  I  don’t  have  a  clear 
understanding  of  how  the  product  looks  when  it  is  considered  to  be  working 
properly.  This  is  a  radical  change  from  what  we  used  to  -  in  the  past,  we 
made  incremental  changes,  but  not  this  time. . . 

Connection  is  too  slow  -  processing  of  data 

Shore  duty  is  rewarded  by  an  online  system.  The  message  will  get  out  that 
you  won’t  be  able  to  access  e-learning  programs  and  online  systems  aboard 
ship. 

We  are  reliant  on  high-level  technology  that  isn’t  currently  available  to  sea 
commands  -  low-ranking  Sailors  do  not  have  access  to  computers  with  which 
to  perform  the  functions  of  the  HPFD/ePerformance  system.  System  is  good 
in  theory,  but  we  aren’t  there  yet.  You  must  find  a  way  to  accommodate  the 
college  graduate  as  well  as  the  naturalized  citizen  (who’s  second  language  is 
English) 

Internet  connection  is  the  big  show  stopper  here. 

Internet  issue  -  there  are  some  things  that  can  be  improved,  but  overall  -  the 
product  is  pretty  good. 
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Internet  Access 


Administrative 

Burden 


Clear 

Instructions  & 
Procedures 


Advantages  to 
NAVFIT98 


Internet  connection  onboard  ship  is  NOT  suitable  to  support 
HPFD/ePerformance . 

System  timing  out  and  problem  understanding  what  I  was  doing  /  wording  of 
within  the  system  make  things  difficult 

I  have  limited  access  (maybe  once  a  week)  to  a  computer.  Even  getting  more 
computer  terminals  wouldn’t  help.  The  time  to  get  on  a  computer  is  limited 
for  shipboard  Sailors. 

More  manning  could  help  free-up  Sailors  and  supervisors  to  get  onto 
computers.  The  mission  of  this  ship  is  to  launch  birds  and  anything  that 
detracts  from  that  doesn’t  work  for  me. 

Amount  of  computer  time  is  going  to  be  a  problem.  I’d  like  to  have  a  paper 
form  that  I  can  hand  to  airmen  to  work  on  when  they  don’t  have  computer 
access. 

A  system  that  requires  a  lot  of  text  to  be  written  could  turn  into  a  “garbage  in; 
garbage  out”  system  in  that  if  a  Sailor  does  not  take  the  time  to  provide  good 
input  -  he  may  receive  a  poor  EVAL  because  the  supervisor  doesn’t  take  the 
time  to  do  a  proper  EVAL  because  he  received  minimal  input  from  junior 
Sailor.  Also,  we  have  other  functions  aside  from  administrative  tasks. 

Maybe  you  could  have  check-boxes  by  the  rating  dimensions.  Less  reliance 
on  writing  text. 

Not  easy  -  would  be  a  C  if  CBT  would  have  been  completed  prior  to  exercise 

Must  know  system  in  order  to  make  it  work  for  you  -  remove  extraneous 
functions  to  bring  it  up  to  a  B  or  an  A  -  incorporate  a  scheduling  system  - 1 
need  to  understand  the  process 

Training  -  explain  how  the  system  works  and  why  we  need  to  perform  the 
required  tasks  -  how  is  the  new  system  different  from  our  current  system? 

No  instructions  for  the  system  -  bugs  in  the  system  -  the  feedback  was  very 
frustrating  since  I  did  not  know  the  system  -  system  could  use  a  Navy 
structure  and  language  to  make  it  easier  for  a  Sailor  to  use 

Needs  better  procedural  instruction  to  make  things  better 

Easier  than  NAVFIT98  - 1  just  want  the  answer 

Grade  A  -  if  everything  gets  fixed  =  grade  F  -  if  today’s  experience 
continues  into  final  product 

Current  NAVFIT98  is  a  chore  -  HPFD/ePerformance  is  much  easier  -  how 
will  this  tie  into  the  big  picture 

Current  system  is  too  easy  to  lose  track  of  deadlines  and  tracking 
accomplishments  (brag  sheet)  -  what  is  the  time-frame  for  these  changes? 

Current  NAVFIT98  is  too  work  intensive  -  HPFD/ePerformance;  much  of 
the  work  is  done  for  you  -  bugs  need  to  be  worked  out  or  the  system  will  be  a 
disaster 
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Product  has  serious  potential. .  .I’d  say  to  go  for  it.  It  can’t  be  worse  than 
NAVFIT98 


Performance 

Appraisal 

Process 


What  are  the  “checks  and  balances”  with  regard  to  document  routing.  Would 
like  to  see  who  is  the  document  holder  in  addition  to  the  document  status. 
Much  like  we  see  with  the  promotion  board  results. 

I’d  like  to  see  more  about  the  process  -  how  the  document  is  routed  -  and  the 
interrelationship  of  the  performance  appraisal  with  other  systems  like 
promotion  boards,  5VM,  advancement  tests,  assignment  process. 


Omit  forced  distribution  -  make  “overall  comments”  block  available  on 
every  ePerformance  document 

Too  early  to  say  -  cannot  evaluate  an  incomplete  product 

System 

Incomplete  Too  early  to  say  -  incomplete  product 

It  needs  work  in  order  to  fix  the  flaws  -  I  need  to  see  more 
Would  give  a  rating  of  “B”  if  HPFD/ePerformance  is  linked/based  on  NKO 
(no  extra  passwords).  The  whole  idea  of  having  a  website  is  to  have  linked 
information.  If  we  have  a  series  of  systems  that  don’t  have  links,  what’s  the 
point. 


System 

Design/Interface 


Fix  the  user-friendliness  issues  (i.e.  the  Menu)  to  make  the  interface  easier  to 
use 


Save/load  documents  to/from  local  disk 


Distinguish  HPFD  from  ePerformance  by  changing  the  color  of  the  form 

Program  does  not  utilize  the  entire  screen  -  it  would  be  better  if  I  could  see 
more  on  the  screen  -  overall  comment  block  should  be  reserved  for  reporting 
senior  comments  or  add  a  reporting  senior  comment  block  - 
HPFD/ePerformance  is  a  great  starting  point 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS  KITTYHAWK 
(CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


C-27 


X. 


Is  there  anything  that  we  haven’t  covered  today  that  you  feel  is  important  to  raise  to 
this  group  for  discussion  of  this  pilot  system? 


Theme 

Performance 

Notes 


Workflow 
Concerns: 
First-level  Rater 
&  Form  Routing 


Implementation 

Issues 


Quotes 

I  really  liked  the  notes  section  . . .  that  was  good. 

I  especially  liked  the  performance  notes  section.  That  way  people  can  quickly 
jot  down  thing  that  might  go  on  their  brag  sheet  before  they  forget. 

How  will  document  routing  work  for  those  for  personnel  external  to  NMCI? 

How  does  the  black  box  work  -  where  do  the  documents  go  when  you  hit 
"complete"? 

Work  center  supervisor  should  be  in  charge  of  his  personnel  and  should  be 
the  one  to  make  recommendations  and  write-up  evaluations. 

I  have  written  evaluations  that  come  back  without  reflecting  my  original 
opinion.  My  words  have  disappeared.  I  like  a  system  that  allows  my 
comments  to  stick. 

Needs  checks  and  balances  to  ensure  supervisor  comments  are  not  ignored  - 
“Reject  /  Submit”  button  (i.e.  “concur/non-concur”) 

Some  folks  are  good  writers  and  we  should  not  expect/force  personnel  to  take 
a  writing  class  just  to  support  the  HPFD/ePerformance  system 

The  work  center  supervisor  has  to  learn  to  be  a  good  writer  eventually.  They 
are  going  to  be  LPO,  CPO,  LCOP  . . .  they’re  going  to  have  to  write  and 
better  to  train  them  early. 

The  average  Sailor  will  not  care  about  failures  in  the  system,  because  it  will 
be  up  to  management  to  figure  things  out  and  make  them  work  -  give  Senior 
Rater  guidance  on  how  to  rate  Sailors  -  Senior  Rater  Average  deviance  will 
help  break  out  Sailors  and  provide  a  level  playing  field  -  limit  comments  to 
top  five  bullets  in  each  comment  box  -  need  reject/concur  box  to  either 
accept  or  kick-back  an  eval 


75%  of  Navy  personnel  transfer  between  June  -  August  (for  various  reasons 
-  kids  going  to  school,  etc)  -  the  current  system  spreads  the  evaluation 
periodicity  over  the  course  of  a  year 


You’ve  got  to  allow  senior  raters  (SCPO)  to  review  a  document  to  “head  off 
a  train  wreck”  before  it  happens  (i.e.  too  many  7.0  ratings  being  submitted) 

Training  should  take  a  minimum  of  6  months  to  educate  the  entire  Navy  on 
new  system  -  make  sure  everyone  knows  the  consequences  of  executing  the 
process  correctly  -  time,  money,  quality  (we'll  get  two  of  these) 

Will  lack  of  "administrative  acumen"  kill  a  Sailor's  career? 

How  is  "deployed"  time  weighted  when  compared  to  "homeport" 
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Concerns  About 

Procedural 

Fairness 


Forced 

Distribution 


HPFD/5VM  seems  to  be  moving  toward  a  “check  list”  rather  than  an 
objective  opinion  of  my  performance  -  what  is  most  important  at  the  end  of 
the  day 

Make  sure  vectors  are  appropriately  weighted  to  ensure  a  person’s 
promotability  is  properly  evaluated  and  the  Sailor  does  not  merely  become  a 
number  to  completes  a  checklist.  Leadership  ability  must  be  taken  seriously 
or  else  junior  Sailors  can  be  poorly  or  mismanaged  by  a  weak/ineffective 
leader. 

How  will  this  fit  into  the  promotion  system? 

Where  does  this  module  exist?  I’d  recommend  it  as  a  stand-alone  program 
within  BupersAccess.  However  it  happens,  the  end  product  must  tie  into 
NKO  for  5VM 

“Good  old  boy  network"  -  new  system  will  come-back  10-fold  as  some  folks 
"beat  the  system" 

Vetting  process  of  ranking  of  Sailors  in  the  "chiefs  mess"  is  removed  -  what 
are  we  losing  in  the  process? 

Will  the  new  system  create  a  "reverse  engineering"  mentality  in  order  to  give 
a  Sailor  a  score  that  will  beat  the  system? 

To  beat  the  culture:  advertise  the  new  system  -  sign  the  instruction  -  need 
visualization  of  document  life-cycle.  A  couple  of  weeks  ago  we  had  a  Master 
Chief  come  up  and  describe  the  process  . . .  that  was  really  helpful. 

CO’s  will  “fudge”  rating  numbers  fit  their  view  of  a  member’s  ranking  within 
their  command  (i.e.  raters  will  still  attempt  to  “rack  &  stack”  within  their 
command). 

When  the  technology  doesn’t  support  the  requirement  -  some  people  might 
feel  that  their  career  (eval)  may  be  rushed  through  -  and  that  being  onboard 
ship  will  hurt  their  career  if  they  cannot  utilize  a  required  resource.  You 
know  you  are  going  to  have  problems  -  so  you  may  pre-complete  a  document 
or  turn  it  in  late 

High  rating  scores  are  expected  since  you  are  penalizing  your  Sailors  for 
being  average  when  other  CO’s  are  not. . .  so  you  inflate  those 

Scores  so  everyone  looks  good  -  khaki  does  not  automatically  afford  a  Sailor 
with  a  specific  level  of  knowledge  and  ability... a  new  chief  doesn’t 
necessarily  know  how  to  rate  a  person’s  performance 

Forced  distribution  has  draw-backs  -  does  not  see  how  the  new  system  will 
highlight  deserving  Sailors  -  who  should  make  Chief  next  year?  How  do  I 
break  out  the  "hot"  Sailor? 

Overview  -  must  be  provided  in  order  for  a  CO  to  make  a  promotion 
recommendation  -  would  like  to  see  “overview”  within  HPFD  tool 

Removing  “forced  distribution”  will  cause  rating  number  creep  - 1  don’t  see 
how  you  can  force  accountability  to  keep  the  numbers  from  creeping  -  limit 
comment  boxes  to  7  or  8  lines  and  the  summary  block  should  be  limited  15  to 
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18  lines  - 


Resistance  to 
Change 


Connectivity 

Concerns 


Getting  rid  of  FD  is  good  since  the  5VM  provides  a  better  model/formula  for 
“racking  &  stacking” 

You  need  FD  for  E6  &  below  because  it  promotes  competitiveness  within  a 
command  and  provides  feedback  to  the  Sailor  saying  “you  are  top-performer 
and  we  think  you  should  be  promoted  early” 

There  is  are  some  ratings  where  a  member’s  qualifications  are  “classified” 
and  cannot  be  displayed  on  their  5VM 

Users  tend  to  wait  for  deadline  -  let's  use  old  system  until  someone  yell's  at 
us.  We  hate  change. 

HPFD  is  new  -  we  fear  change  -  let’s  use  the  old  system,  because  it  works  - 
the  majority  will  wait  until  the  last  minute 

Folks  are  having  serious  connection  /  timeout  problems  with  NKO  -  training 
will  be  required  to  get  people  used  to  the  new  system  -  new  folks  will  have 
trouble  filling  up  comment  boxes  - 

Currently  NAVFIT98  is  maintained  on  the  local  LAN  and  then  transmitted 
once  the  boat  pulls  into  port  -  you  won’t  be  able  to  sell  the  idea  of  “not  being 
able  to  change  a  document  in  midstream”  -  dropping  an  80%  solution  on  the 
fleet  without  education  will  be  a  BIG  mistake 

As  connectivity  capability  increases,  we  will  be  better  able  to 
transmit/receive  at  sea 

At  sea  connectivity  will  require  an  UNCLAS  net  -  system  must  be  NMCI 
compliant  -  on-line  Performance  notes  are  a  good  thing  and  will  keep  you 
from  forgetting  daily  accomplishments  -  the  document  must  be  a  living 
document  in  that  you  must  be  able  to  continuously  chop  a  document  until  it  is 
done  -  training/documentation  MUST  be  in  place  long  before  the  fleet  will 
be  able  to  utilize  an  80%  solution 


ZIP  files  are  not  allowed  by  NMCI  email  system 

Will  require  a  local  server  onboard  to  be  able  to  “push/pull”  while  at  sea 

Miscellaneous  Who  gets  to  read  your  performance  notes?  Junior  personnel  are  sometimes 
grammatically  challenged  and  will  tend  to  over-inflate  their  own 
performance. 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS  KITTYHAWK 
(CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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Appendix  D 

Non-supervisory  Personnel: 

Main  Content  Themes  and  Comments 


II.  Starting  with  the  briefing  that  described  the  new  HPFD  and  ePerformance  plan.  This 
was  designed  to  give  you  a  good  overview  of  the  new  system  and  what  features 
changed  from  our  previous  performance  appraisal  system. 

c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 

None 


d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 

Quotes 

The  briefing  needed  to  be  condensed,  more  direct,  to  the  point,  less 
complicated.  Focus  on  the  tasks  that  you’re  going  to  be  doing. 

Save  the  discussion  of  features  for  the  end. 

Question:  will  the  tutorial  be  part  of  the  system/instruction?  Note:  The 
comments  for  the  briefing  text  should  be  considered  when  writing  the 
instruction  -  concise,  to-the-point,  simple,  easy  to  follow. 


Theme 

Clarity 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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m.  Now  we’ll  talk  about  training  and  reference  materials.  These  materials  were  designed 
to  help  familiarize  you  with  the  system  and  get  you  to  the  documents  necessary  to 
conduct  your  HPFD  and  ePerformance  session  [SOME  PARTICIPANTS  WILL 
RECEIVE  A  HAND-CARD,  OTHERS  WILL  COMPLETE  THE  CBT] 

c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 

Quick  Reference  Guide 

Quotes 

The  write-up  described  to  system  concepts  well 

The  QRG  is  more  valuable  than  the  tutorial  -  if  I  had  to  choose,  based  on 
“time  availability”  with  a  PC,  I  would  choose  a  QRG,  because  I  don’t  have 
time  to  sit  through  a  tutorial 

QRG  should  be  called  an  “Information  Map” 

“Quick  reference”  guide  is  outstanding  -  simple  guidance  is  better  in  that 
you  get  clear,  concise  steps  to  perform  a  task 

After  receiving  the  quick  reference  guide  - 1  could  have  written  a  HPFD 
document  without  a  problem 

4  of  4  used  QRG 

Used  QRG  to  find  documents  and  buttons 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


Computer-based  Training: 

Theme  Quotes 

Clarity  CBT  was  laid  out  in  logical  manner 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


Theme 

General 

Satisfaction 
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d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 

Quick  Reference  Guide: 

Theme  Quotes 

Include  Icons  in  Include  a  list  of  the  icons  and  what  they  stand  for. 

QRG 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


Computer-based  Training: 

Theme  Quotes 

Clarity  of  Wording  of  the  tutorial  could  be  better 

Information 

Too  many  abbreviations  to  memorize  at  once 

I  seemed  to  keep  reading  the  same  thing  over  and  over  -  the  tutorial  was  too 
confusing  -  should  be  able  to  navigate  the  tutorial  to  find  specific 
information  rather  than  try  to  cram  the  entire  text  into  memory 

I  need  a  better  overview  before  I  just  go  into  the  features  of  the  system 

Make  the  instruction  more  concise  -  shorten  the  instruction  where  possible 

Getting  through  the  process  the  first  time  is  difficult 

It  took  a  while  to  grasp  the  concepts  being  presented  and  required  some 
coaching 

I  don’t  necessarily  need  to  know  why  I  press  a  button,  just  tell  me  what  to 
do 

CBT  was  overwhelming  -  too  much  information  being  put-out 
CBT  not  logical 

Learning  objectives  not  clear  -  need  to  simplify  CBT 

Too  many  slides  with  too  much  information  -  break  down  information  into 
logical  sections  -  integrate  CBT  with  “hands-on”  to  reinforce  learning  or 
have  it  integrated  with  the  system  (e.g.,  links  from  CBT  to  live  system) 

Need  to  add  “prep”  -  what  are  the  learning  objectives? 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KiTTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


Simplicity  of 

Learning 

Process/Didn’t 

Relay  Document 

Navigation 

information 
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IV.  Starting  with  logging  onto  the  system.  You  needed  to  do  several  tasks  -  enter  a 

username,  password,  and  select  the  Navy  Performance  Management  and  Appraisal 
system. 

a.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 

Theme  Quotes 

Ease  Of  Login  When  Logging  in  is  straight  forward 

Functioning 

Changing  my  password  worked  well 
Logged  in  by  the  staff 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


b.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Password 

Requirements 


Quotes 

Password  reminder  would  be  helpful  for  those  folks  who  tend  to 
forget/misplace  passwords 

System  is  lacking  guidance  for  acceptable  password  entries  when 
“changing  your  password” 

Logging  on  took  a  long  time 

4  of  4  had  concerns  with  the  number  of  passwords  they  needed  to 
have 

Either  make  the  system  so  that  you  log  in  once  and  have  access  to  all 
systems  (e.g.,  MyPay,  NKO,  BOL,  NMCI)  or  make  the  password 
requirements  (one  upper  case,  one  lower  case,  one  alphanumeric,  etc) 
the  same  so  one  password  would  work  for  each  system 

Use  SSN  (already  being  used  by  NKO,  MyPay,  and  BOL)  -  as  long 
as  system  is  “secure”  -  If  someone  wanted  to  steal  your  identity,  there 
are  a  lot  easier  ways  of  getting  it  than  breaking  into  a  government 
website 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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V. 


Now  let’s  discuss  selecting  and  opening  your  document.  Here  you  are  asked  to 
identify  the  appropriate  document,  select,  and  open  your  document  awaiting  action. 

c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 

Quotes 

Once  you  get  to  the  document  folder,  the  document  was  easy  to  find  - 
there  was  no  guessing  involved  as  long  as  the  QRG  was  available  -  the 
tutorial  combined  with  the  QRG  make  the  process  VERY  easy  to 
understand. 

4  of  4  -  used  the  quick  reference  guide  to  navigate  the  system 

Really  didn’t  pay  too  much  attention  to  the  PeopleSoft  terms  on  the  left 
side  of  the  document  console  because  they  went  right  to  the  QRG. 

I  had  to  refer  back  to  the  QRG  to  help  me  find  the  document  folder 

This  is  where  I  used  my  QRG. 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


Theme 

Ease  of  Selecting 
Document 
Facilitated  by 
Quick  Reference 
Guide 


d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Trouble  Finding 
the  Document  - 
Identification 
Trouble  Finding 
the  Document  - 
Navigation 


Quotes 

There  is  no  clear  initial  indication  as  to  where  documents  are  stored. 
There  is  too  much  information  on  the  left  side  of  the  screen 


Did  not  know  the  contents  of  each  menu  item  -  had  to  perform  search  in 
order  to  find  the  HPFD  document 

Guidelines  -  session  path  -  taskbar  buttons  for  major  tasks 

I  clicked  through  every  option  on  the  left  side  before  I  found  the 
performance  document 

I  wished  I  had  a  QRG  (Note:  Participant  had  a  QRG  but  did  not  refer  to 
it) 


Breakdown  functions  more  logically 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KittYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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Let’s  talk  about  completing  (filling  in)  your  HPFD  form.  Let’s  list  the  aspects  of 
completing  the  form  -  filling  in  the  background  information,  completing  the 
performance  dimension,  finalizing  the  form... 


c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Performance 

Dimensions 

Quotes 

Performance  dimensions  were  well  thought  out  and  easy  to  understand  - 1 
like  them  better  than  the  current  system  -  the  new  dimensions  more  easily 
lend  themselves  to  rating  performance  on  “everyday”  Navy  jobs 

Behavioral  descriptions  worked  well.  Terminology  was  not  too  civilian  and 
well  understood  by  all. 

Web-based 

Form 

Text  box  should  be  available  to  keep  accomplishments  from  fading  from 
memory 

I  like  being  able  to  select  an  “honest”  appraisal  of  my  performance  with  the 
option  to  map  a  path  to  improve  where  you  are  weak  - 1  like  the  “text  box” 
associated  with  each  behavioral  category 

Process 

Ownership 

I  like  being  able  to  provide  input  -  “counseling”  is  typically  an  afterthought 
and  generally  a  “pencil  whip”  document  that  has  no  value  and  give  no 
performance  redirection 

Useful  - 1  can  focus  on  the  areas  where  I  want  to  improve  without  being 
distracted  by  extraneous  behavioral  traits 

HPFD  will  tie  into  5VM  and  help  improve  a  Sailor 

General  Positive 

Once  you  find  the  document  -  there  is  not  whole  lot  to  it.  This  was  the  easy 
part  -  the  hard  part  was  finding  the  document. 

Performance 

Notes 

I  like  the  IDP  functionality  and  Performance  notes 

“Performance  notes”  is  a  great  idea.  Right  now  we  keep  accomplishments  in 
a  green  notebook.  It  would  be  nice  to  be  able  to  document  those  at  the  end 
of  the  day  or  week. 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 


d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

HPFD  Process 

Quotes 

Don’t  document  sensitive  items  -  some  things  should  stay  between  the 
supervisor  and  the  Sailor.  We’d  discuss  those  things  informally  or  put  it  on 
a  counseling  sheet  first. 

All:  1)  so  the  supervisor  could  provide  their  input  and  have  something  for 
the  Sailor  to  review  face-to-face  first,  make  any  necessary  revisions,  then 
finalize  with  a  printed  form  that  Sailors  sign,  and  2)  in  the  event  the  server  is 
down  or  you  don’t  have  Internet  connectivity  because  you’re  on  patrol. 

Document  Back¬ 
up:  Paper 
Copy/Printable 
Version 
Internet/NSIPS 
Connectivity 

All  -  wanted  a  printed  document 

For  the  off -crew,  a  web-based  system  would  work  fine.  Nearly  everyone  has 
the  capability  of  getting  on  the  Internet. 

Currently,  we  get  as  much  admin  work  done  as  possible  prior  to  going  on 
patrol.  Otherwise,  we  work  from  a  printed  form  and  input  it  once  we  return. 

To  make  this  work  aboard  submarines  on  patrol,  you’d  need  a  NSIPS  server 
aboard  the  sub  that  people  could  work  from.  Once  the  sub  comes  in,  then 
there’d  need  to  be  a  way  to  connect  that  server  to  the  Internet,  since  we 
currently  don’t  have  non-secure  connections  for  the  Internet.  All  of  our 
computers  aboard  the  sub  are  secret/classified. 

Implementation 

Concerns 

Too  much  time/information  for  HPFD  - 

The  Fleet  would  need  a  period  of  training  to  use  the  system  and  become 
familiar  with  it  before  it  went  live  -  like  the  Fleet  beta  testing  we’re  using 
for  the  FSM  (financial  management  system)  prototype  we’re  using. 

Miscellaneous 

“Spell  Check”  button  should  be  easier  to  find/identify  (e.g.  create  a  button 
with  the  word  “spell  check”) 

It  mieht  help  to  have  examples  like  “Completed  man-hours  of  in 

this  period.” 

Behavioral  traits  are  well  fleshed  out  -  trait  text  should  remain  separate  in  a 
printed  format 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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VII.  Now,  let’s  talk  about  completing  (filling  in)  your  ePerformance  form.  Some  of  you 
will  complete  a  document  for  yourselves  as  the  employee  while  others  will  be 
working  on  documents  for  those  who  work  for  you. 

c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 
N/A 

d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 

N/A 


D-8 


VIII.  Let’s  talk  about  wrapping  up  or  closing  your  HPFD  or  performance  appraisal  session. 


c.  Which  aspects  worked  well  -  what  about  it  made  you  feel  that  it  worked  well? 


Theme 

Performance 

Notes 

Quotes 

“Performance  notes”  is  a  good  place  to  put  comments  about  daily 
performance  accomplishments  between  HPFD/ePerformance  sessions.  “I 
can  put  down  my  OJT  and  other  things  I  do  on  my  job  there  to  remember.” 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 

d.  Which  aspects  could  stand  improvement  -  what  about  it  didn’t  work  for  you  and 
what  would  help? 


Theme 

Save  and 

Complete/ 

Forward 

Document 

Function 

Quotes 

“Complete”  button  is  too  easy  to  confuse  with  “save”  button  -  “complete” 
button  needs  a  [confirmation  popup] 

Re-label  the  “complete”  button  to  avoid  confusion  as  well  as  routing 
incomplete  documents 

Performance  notes  should  have  been  one  of  the  first  options  you  could  pick 
from 

Combine  HPFD  document  and  Performance  Notes  documents 

Back-up 

Copies/Printable 

Version 

What  kind  of  contingency  plan  is  there  -  what  if  the  HPFD/ePerformance 
system  fails  (virus,  server  crash,  etc)?  You’d  need  a  paper  backup. 

Having  a  paper  copy  would  allow  you  to  print  documents  off  and  bring 
them  around  for  pen  and  ink  changes  and  enter  them  into  the  online 
document  when  you  have  access. 

HPFD  Process 

Face-to-face  will  cut  out  “document  ping-pong” 

ALL  -  host  HPFD/Performance  Notes  system  on  Ship’s  Intranet 

System  Access 

Online  time  is  difficult  to  come-by. 

Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KjttYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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If  you  had  to  assign  a  grade  to  this  system,  what  grade  would  you  assign? 
(A=Outstanding,  C=Average,  F=Failed) 

c.  What  grade  would  you  assign  to  this  pilot  system?  What  led  you  to  assign  this 
grade? 

d.  What,  if  anything,  could  be  done  to  cause  you  to  raise  your  grade  of  this  system? 

Group  Current  GPA 

NAS  Brunswick  2.0 

Brunswick,  ME 

USS  KITTYHAWK  (CY63)  2.17 

Yokosuka,  Japan 

Kitsap  Naval  Base  -  Bangor  2.5 

Bangor,  WA 


Theme 

Clear 

Instructions  & 
Procedures 


Quotes 

It  was  a  D  until  I  got  used  to  the  system.  Need  contextually  based  tutorial 
so  I  can  look  up  specific  information  about  the  system. 

The  QRG  would  bring  the  score  up  to  an  “A” 

System/tutorial  needs  to  be  concise  because  of  high-tempo  onboard 

Making  the  system  easier  and  more  simplified  would  bring  the  score  up  to 
an  “A” 


NSIPS/ 

Connectivity 

Problems 


Performance 

Appraisal 

Process 


The  system  seemed  adequate,  but  the  system  needs  to  be  fixed  -  provide 
better  tools  to  understand  system  -  troubleshoot  and  fix  areas  of  confusion 
(see  notes  above) 

With  the  bugs  currently  encountered 
Not  a  complete  product 

Process  is  too  loose  -  if  system  becomes  a  1-2-3  process  - 

Less  computer  time  -  reduce  reliance  on  PC  to  perform  the  HPFD  function. 
Paper-copy  back-up  will  help  facilitate  face-to-face  discussions. 

Spend  more  face-to-face  time  instead  of  sending  emails  back  and  forth. 

Cut  out  some  of  the  process  -  there  is  too  much  time  being  spent  on  the 
computer  -  reduce  the  “ping  pong”  so  the  Sailor  can  be  productive  instead 
of  sitting  at  a  computer. 

Streamline  the  process.  We  need  to  reduce  the  amount  of  time  we’re  on  the 
computer  and  in  order  to  spend  time  doing  our  jobs. 


General 


The  hardest  thing  was  getting  into  the  document,  the  rest  was  easy 


Satisfaction 


The  document  was  easy  to  use. 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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Is  there  anything  that  we  haven’t  covered  today  that  you  feel  is  important  to  raise  to 
this  group  for  discussion  of  this  pilot  system? 


Theme 

HPFD  Appraisal 
Process 


NSIPS/Internet 

Connectivity 


System  Training 
Concerns 


Quotes 

Fill  out  a  paper  HPFD,  do  a  face-to-face,  then  go  back  and  enter  the 
document  into  the  computer 

Complete  HPFD  &  route  to  member,  member  does  self-assessment,  do  face- 
to-face  with  member,  create  IDP,  finalize  the  document 

Supervisor  provides  inputs  to  subordinate  -  do  self-assessment  -  face-to- 
face  -  finalize 

Those  with  non-administrative  jobs  did  not  like  the  need  to  route  documents 
via  email/system  to  review,  approve,  edit,  review,  approve.  Too  much  work 
and  will  require  everyone  to  be  continually  checking  their  email  -  But,  if 
that’s  what  Navy  leadership  requires,  we’ll  have  to  get  onboard. 

Do  not  like  the  idea  of  a  distributed  workload  -  E3/4/5  will  not  necessarily 
know  how  to  effectively  evaluate/mentor  someone  below  them 

NKO  -  Question  about  how  NSIPS  will  work  aboard  ship  if  within  NKO  if 
ships  don’t  have  NKO.  Have  a  NKO  server  aboard  ship  with  NSIPS  and  do 
a  push/pull  when  updates  and  information  bursts  are  necessary. 

NKO  would  be  a  reasonable  location  for  HPFD/ePerformance  system 

System  needs  to  be  implemented  onboard  -  since  internet  connectivity  is  not 
easily  maintained 

CBT  -  needs  an  index  of  terms  for  help  on  specific  topics  rather  than  going 
through  an  entire  chapter  of  the  tutorial  to  get  refresher  training  on  how  do 
accomplish  one  specific  task 

Put  the  QRG  as  a  resource  document  on  the  website. 


Note:  Participant  Location  Code  is  as  follows:  NAS  Brunswick,  Brunswick,  ME;  USS 
KITTYHAWK  (CV63),  Yokosuka,  Japan;  Kitsap  Naval  Base  -  Bangor,  Bangor,  WA. 
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